<html><head>
      <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
   <title>Apache OpenNLP Developer Documentation</title><link rel="stylesheet" href="css/opennlp-docs.css" type="text/css"><meta name="generator" content="DocBook XSL-NS Stylesheets V1.75.2"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div lang="en" class="book" title="Apache OpenNLP Developer Documentation"><div class="titlepage"><div><div><h1 class="title"><a name="d4e1"></a>Apache OpenNLP Developer Documentation</h1></div><div><div class="authorgroup">
			<h3 class="corpauthor">Written and maintained by the Apache OpenNLP Development
				Community</h3>
		</div></div><div><p class="releaseinfo">
			Version 1.5.2-incubating
		</p></div><div><p class="copyright">Copyright &copy; ,  The Apache Software Foundation</p></div><div><div class="legalnotice" title="Legal Notice"><a name="d4e7"></a>
			<p title="License and Disclaimer">
				<b>License and Disclaimer.&nbsp;</b>
				
					The ASF licenses this documentation
					to you under the Apache License,
					Version 2.0 (the
					"License"); you may not use this documentation
					except in compliance
					with the License. You may obtain a copy of the
					License at

					</p><div class="blockquote"><blockquote class="blockquote">
						<p>
							<a class="ulink" href="http://www.apache.org/licenses/LICENSE-2.0" target="_top">http://www.apache.org/licenses/LICENSE-2.0</a>
						</p>
					</blockquote></div><p title="License and Disclaimer">

					Unless required by applicable law or agreed to in writing,
					this documentation and its contents are distributed under the License
					on an
					"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
					KIND, either express or implied. See the License for the
					specific language governing permissions and limitations
					under the License.
				
			</p>
		</div></div><div><p class="pubdate">
			, 
		</p></div></div><hr></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="chapter"><a href="#opennlp">1. Introduction</a></span></dt><dt><span class="chapter"><a href="#tools.sentdetect">2. Sentence Detector</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.sentdetect.detection">Sentence Detection</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.sentdetect.detection.cmdline">Sentence Detection Tool</a></span></dt><dt><span class="section"><a href="#tools.sentdetect.detection.api">Sentence Detection API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.sentdetect.training">Sentence Detector Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.sentdetect.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.sentdetect.training.tool">Training API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.sentdetect.eval">Evaluation</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.sentdetect.eval.tool">Evaluation Tool</a></span></dt></dl></dd></dl></dd><dt><span class="chapter"><a href="#tools.tokenizer">3. Tokenizer</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.tokenizer.introduction">Tokenization</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.tokenizer.cmdline">Tokenizer Tools</a></span></dt><dt><span class="section"><a href="#tools.tokenizer.api">Tokenizer API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.tokenizer.training">Tokenizer Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.tokenizer.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.tokenizer.training.api">Training API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.tokenizer.detokenizing">Detokenizing</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.tokenizer.detokenizing.api">Detokenizing API</a></span></dt><dt><span class="section"><a href="#tools.tokenizer.detokenizing.dict">Detokenizer Dictionary</a></span></dt></dl></dd></dl></dd><dt><span class="chapter"><a href="#tools.namefind">4. Name Finder</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.namefind.recognition">Named Entity Recognition</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.namefind.recognition.cmdline">Name Finder Tool</a></span></dt><dt><span class="section"><a href="#tools.namefind.recognition.api">Name Finder API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.namefind.training">Name Finder Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.namefind.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.namefind.training.api">Training API</a></span></dt><dt><span class="section"><a href="#tools.namefind.training.featuregen">Custom Feature Generation</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.namefind.eval">Evaluation</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.namefind.eval.tool">Evaluation Tool</a></span></dt><dt><span class="section"><a href="#tools.namefind.eval.api">Evaluation API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.namefind.annotation_guides">Named Entity Annotation Guidelines</a></span></dt></dl></dd><dt><span class="chapter"><a href="#tools.doccat">5. Document Categorizer</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.doccat.classifying">Classifying</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.doccat.classifying.cmdline">Document Categorizer Tool</a></span></dt><dt><span class="section"><a href="#tools.doccat.classifying.api">Document Categorizer API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.doccat.training">Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.doccat.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.doccat.training.api">Training API</a></span></dt></dl></dd></dl></dd><dt><span class="chapter"><a href="#tools.postagger">6. Part-of-Speech Tagger</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.postagger.tagging">Tagging</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.postagger.tagging.cmdline">POS Tagger Tool</a></span></dt><dt><span class="section"><a href="#tools.postagger.tagging.api">POS Tagger API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.postagger.training">Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.postagger.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.postagger.training.api">Training API</a></span></dt><dt><span class="section"><a href="#tools.postagger.training.tagdict">Tag Dictionary</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.postagger.eval">Evaluation</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.postagger.eval.tool">Evaluation Tool</a></span></dt></dl></dd></dl></dd><dt><span class="chapter"><a href="#tools.chunker">7. Chunker</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.parser.chunking">Chunking</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.parser.chunking.cmdline">Chunker Tool</a></span></dt><dt><span class="section"><a href="#tools.parser.chunking.api">Chunking API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.chunker.training">Chunker Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.chunker.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.chunker.training.api">Training API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.chunker.evaluation">Chunker Evaluation</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.chunker.evaluation.tool">Chunker Evaluation Tool</a></span></dt></dl></dd></dl></dd><dt><span class="chapter"><a href="#tools.parser">8. Parser</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.parser.parsing">Parsing</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.parser.parsing.cmdline">Parser Tool</a></span></dt><dt><span class="section"><a href="#tools.parser.parsing.api">Parsing API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.parser.training">Parser Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.parser.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.parser.training.api">Training API</a></span></dt></dl></dd></dl></dd><dt><span class="chapter"><a href="#tools.coref">9. Coreference Resolution</a></span></dt><dt><span class="chapter"><a href="#tools.corpora">10. Corpora</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.corpora.conll">CONLL</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.corpora.conll.2000">CONLL 2000</a></span></dt><dt><span class="section"><a href="#tools.corpora.conll.2003">CONLL 2002</a></span></dt><dt><span class="section"><a href="#tools.corpora.conll.2003">CONLL 2003</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.corpora.arvores-deitadas">Arvores Deitadas</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.corpora.arvores-deitadas.getting">Getting the data</a></span></dt><dt><span class="section"><a href="#tools.corpora.arvores-deitadas.converting">Converting the data</a></span></dt><dt><span class="section"><a href="#tools.corpora.arvores-deitadas.evaluation">Evaluation</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.corpora.leipzig">Leipzig Corpora</a></span></dt></dl></dd><dt><span class="chapter"><a href="#opennlp.ml">11. Machine Learning</a></span></dt><dd><dl><dt><span class="section"><a href="#opennlp.ml.maxent">Maximum Entropy</a></span></dt><dd><dl><dt><span class="section"><a href="#opennlp.ml.maxent.impl">Implementation</a></span></dt></dl></dd></dl></dd><dt><span class="chapter"><a href="#org.apche.opennlp.uima">12. UIMA Integration</a></span></dt><dd><dl><dt><span class="section"><a href="#org.apche.opennlp.running-pear-sample">Running the pear sample in CVD</a></span></dt><dt><span class="section"><a href="#org.apche.opennlp.further-help">Further Help</a></span></dt></dl></dd></dl></div><div class="list-of-tables"><p><b>List of Tables</b></p><dl><dt>4.1. <a href="#d4e205">Genertor elements</a></dt></dl></div>
	

	
	
	<div class="chapter" title="Chapter&nbsp;1.&nbsp;Introduction"><div class="titlepage"><div><div><h2 class="title"><a name="opennlp"></a>Chapter&nbsp;1.&nbsp;Introduction</h2></div></div></div>

<p>
The Apache OpenNLP library is a machine learning based toolkit for the processing of natural language text.
It supports the most common NLP tasks, such as tokenization, sentence segmentation,
part-of-speech tagging, named entity extraction, chunking, parsing, and coreference resolution.
These tasks are usually required to build more advanced text processing services.
OpenNLP also includes maximum entropy and perceptron based machine learning.
</p>

<p>
The goal of the OpenNLP project will be to create a mature toolkit for the abovementioned tasks.
An additional goal is to provide a large number of pre-built models for a variety of languages, as
well as the annotated text resources that those models are derived from.
</p>
</div>
	<div class="chapter" title="Chapter&nbsp;2.&nbsp;Sentence Detector"><div class="titlepage"><div><div><h2 class="title"><a name="tools.sentdetect"></a>Chapter&nbsp;2.&nbsp;Sentence Detector</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#tools.sentdetect.detection">Sentence Detection</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.sentdetect.detection.cmdline">Sentence Detection Tool</a></span></dt><dt><span class="section"><a href="#tools.sentdetect.detection.api">Sentence Detection API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.sentdetect.training">Sentence Detector Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.sentdetect.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.sentdetect.training.tool">Training API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.sentdetect.eval">Evaluation</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.sentdetect.eval.tool">Evaluation Tool</a></span></dt></dl></dd></dl></div>

	

	<div class="section" title="Sentence Detection"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.sentdetect.detection"></a>Sentence Detection</h2></div></div></div>
		
		<p>
		The OpenNLP Sentence Detector can detect that a punctuation character 
		marks the end of a sentence or not. In this sense a sentence is defined 
		as the longest white space trimmed character sequence between two punctuation
		marks. The first and last sentence make an exception to this rule. The first 
		non whitespace character is assumed to be the begin of a sentence, and the 
		last non whitespace character is assumed to be a sentence end.
		The sample text below should be segmented into its sentences.
		</p><pre class="programlisting">
				
Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29. Mr. Vinken is
chairman of Elsevier N.V., the Dutch publishing group. Rudolph Agnew, 55 years
old and former chairman of Consolidated Gold Fields PLC, was named a director of this
British industrial conglomerate.
		</pre><p>
		After detecting the sentence boundaries each sentence is written in its own line.
		</p><pre class="programlisting">
				
Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29.
Mr. Vinken is chairman of Elsevier N.V., the Dutch publishing group.
Rudolph Agnew, 55 years old and former chairman of Consolidated Gold Fields PLC,
    was named a director of this British industrial conglomerate.
		</pre><p>
		Usually Sentence Detection is done before the text is tokenized and thats the way the pre-trained models on the web site are trained,
		but it is also possible to perform tokenization first and let the Sentence Detector process the already tokenized text.
		The OpenNLP Sentence Detector cannot identify sentence boundaries based on the contents of the sentence. A prominent example is the first sentence in an article where the title is mistakenly identified to be the first part of the first sentence.
		Most components in OpenNLP expect input which is segmented into sentences.
		</p>
		
		<div class="section" title="Sentence Detection Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.sentdetect.detection.cmdline"></a>Sentence Detection Tool</h3></div></div></div>
		
		<p>
		The easiest way to try out the Sentence Detector is the command line tool. The tool is only intended for demonstration and testing.
		Download the english sentence detector model and start the Sentence Detector Tool with this command:
				</p><pre class="programlisting">
				
$bin/opennlp SentenceDetector en-sent.bin
		</pre><p>
		Just copy the sample text from above to the console. The Sentence Detector will read it and echo one sentence per line to the console.
		Usually the input is read from a file and the output is redirected to another file. This can be achieved with the following command.
		</p><pre class="programlisting">
				
$bin/opennlp SentenceDetector en-sent.bin &lt; input.txt &gt; output.txt
		</pre><p>
		For the english sentence model from the website the input text should not be tokenized.
		</p>
		</div>
		<div class="section" title="Sentence Detection API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.sentdetect.detection.api"></a>Sentence Detection API</h3></div></div></div>
		
		<p>
		The Sentence Detector can be easily integrated into an application via its API.
		To instantiate the Sentence Detector the sentence model must be loaded first.
		</p><pre class="programlisting">
				
InputStream modelIn = <b class="hl-keyword">new</b> FileInputStream(<b class="hl-string"><i style="color:red">"en-sent.bin"</i></b>);

<b class="hl-keyword">try</b> {
  SentenceModel model = <b class="hl-keyword">new</b> SentenceModel(modelIn);
}
<b class="hl-keyword">catch</b> (IOException e) {
  e.printStackTrace();
}
<b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (modelIn != null) {
    <b class="hl-keyword">try</b> {
      modelIn.close();
    }
    <b class="hl-keyword">catch</b> (IOException e) {
    }
  }
}
		</pre><p>
		After the model is loaded the SentenceDetectorME can be instantiated.
		</p><pre class="programlisting">
				
SentenceDetectorME sentenceDetector = <b class="hl-keyword">new</b> SentenceDetectorME(model);
		</pre><p>
		The Sentence Detector can output an array of Strings, where each String is one sentence.
				</p><pre class="programlisting">
				
String sentences[] = sentenceDetector.sentDetect(<b class="hl-string"><i style="color:red">"  First sentence. Second sentence. "</i></b>);
		</pre><p>
		The result array now contains two entires. The first String is "First sentence." and the second String is "Second sentence." The whitespace before, between and after the input String is removed.
		The API also offers a method which simply returns the span of the sentence in the input string.
		</p><pre class="programlisting">
				
Span sentences[] = sentenceDetector.sentPosDetect(<b class="hl-string"><i style="color:red">"  First sentence. Second sentence. "</i></b>);
		</pre><p>
		The result array again contains two entires. The first span beings at index 2 and ends at 17. The second span begins at 18 and ends at 34. The utility method Span.getCoveredText can be used to create a substring which only covers the chars in the span.
		</p>
		</div>
	</div>
	<div class="section" title="Sentence Detector Training"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.sentdetect.training"></a>Sentence Detector Training</h2></div></div></div>
		
		<p></p>
		<div class="section" title="Training Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.sentdetect.training.tool"></a>Training Tool</h3></div></div></div>
		
		<p>
		OpenNLP has a command line tool which is used to train the models available from the model
		download page on various corpora. The data must be converted to the OpenNLP Sentence Detector
		training format. Which is one sentence per line. An empty line indicates a document boundary.
		In case the document boundary is unknown, its recommended to have an empty line every few ten
		sentences. Exactly like the output in the sample above.
		Usage of the tool:
		</p><pre class="programlisting">
				
$ bin/opennlp SentenceDetectorTrainer
Usage: opennlp SentenceDetectorTrainer [-abbDict path] [-params paramsFile] -lang language \
[-cutoff num] [-iterations num] [-encoding charsetName] -data trainData -model modelFile
Arguments description:
	-abbDict path
		The abbreviation dictionary in XML format.
	-params paramsFile
		Training parameters file.
	-lang language
		specifies the language which is being processed.
	-cutoff num
		specifies the min number of times a feature must be seen. It is ignored if a parameters file is passed.
	-iterations num
		specifies the number of training iterations. It is ignored if a parameters file is passed.
	-encoding charsetName
		specifies the encoding which should be used for reading and writing text. If not specified the system default will be used.
	-data trainData
		the data to be used during training
	-model modelFile
		the output model file
		</pre><p>
		To train an English sentence detector use the following command:
				</p><pre class="programlisting">
				
$bin/opennlp SentenceDetectorTrainer -encoding UTF-8 -lang en -data en-sent.train -model en-sent.bin

Indexing events using cutoff of 5

	Computing event counts...  done. 4883 events
	Indexing...  done.
Sorting and merging events... done. Reduced 4883 events to 2945.
Done indexing.
Incorporating indexed data for training...  
done.
	Number of Event Tokens: 2945
	    Number of Outcomes: 2
	  Number of Predicates: 467
...done.
Computing model parameters...
Performing 100 iterations.
  1:  .. loglikelihood=-3384.6376826743144	0.38951464263772273
  2:  .. loglikelihood=-2191.9266688597672	0.9397911120212984
  3:  .. loglikelihood=-1645.8640771555981	0.9643661683391358
  4:  .. loglikelihood=-1340.386303774519	0.9739913987302887
  5:  .. loglikelihood=-1148.4141548519624	0.9748105672742167

 ...&lt;skipping a bunch of iterations&gt;...

 95:  .. loglikelihood=-288.25556805874436	0.9834118369854598
 96:  .. loglikelihood=-287.2283680343481	0.9834118369854598
 97:  .. loglikelihood=-286.2174830344526	0.9834118369854598
 98:  .. loglikelihood=-285.222486981048	0.9834118369854598
 99:  .. loglikelihood=-284.24296917223916	0.9834118369854598
100:  .. loglikelihood=-283.2785335773966	0.9834118369854598
Wrote sentence detector model.
Path: en-sent.bin

		</pre><p>
		</p>
		</div>
		<div class="section" title="Training API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.sentdetect.training.tool"></a>Training API</h3></div></div></div>
		
		<p>
		The Sentence Detector also offers an API to train a new sentence detection model.
		Basically three steps are necessary to train it:
		</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
					<p>The application must open a sample data stream</p>
				</li><li class="listitem">
					<p>Call the SentenceDetectorME.train method</p>
				</li><li class="listitem">
					<p>Save the SentenceModel to a file or directly use it</p>
				</li></ul></div><p>
			The following sample code illustrates these steps:
					</p><pre class="programlisting">
				
Charset charset = Charset.forName(<b class="hl-string"><i style="color:red">"UTF-8"</i></b>);				
ObjectStream&lt;String&gt; lineStream = <b class="hl-keyword">new</b> PlainTextByLineStream(<b class="hl-keyword">new</b> FileInputStream(<b class="hl-string"><i style="color:red">"en-sent.train"</i></b>),
    charset);
ObjectStream&lt;SentenceSample&gt; sampleStream = <b class="hl-keyword">new</b> SentenceSampleStream(lineStream);

SentenceModel model;

<b class="hl-keyword">try</b> {
  model = SentenceDetectorME.train(<b class="hl-string"><i style="color:red">"en"</i></b>, sampleStream, true, null, <span class="hl-number">5</span>, <span class="hl-number">100</span>);
}
<b class="hl-keyword">finally</b> {
  sampleStream.close();
}

OutputStream modelOut = null;
<b class="hl-keyword">try</b> {
  modelOut = <b class="hl-keyword">new</b> BufferedOutputStream(<b class="hl-keyword">new</b> FileOutputStream(modelFile));
  model.serialize(modelOut);
} <b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (modelOut != null) 
     modelOut.close();      
}
		</pre><p>
		</p>
		</div>
	</div>
	<div class="section" title="Evaluation"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.sentdetect.eval"></a>Evaluation</h2></div></div></div>
		
		<p>
		</p>
		<div class="section" title="Evaluation Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.sentdetect.eval.tool"></a>Evaluation Tool</h3></div></div></div>
			
			<p>
				The command shows how the evaluator tool can be run:
				</p><pre class="programlisting">
				
$bin/opennlp SentenceDetectorEvaluator -encoding UTF-8 -model en-sent.bin -data en-sent.eval  

Loading model ... done
Evaluating ... done

Precision: 0.9465737514518002
Recall: 0.9095982142857143
F-Measure: 0.9277177006260672
				</pre><p>
				The en-sent.eval file has the same format as the training data.
			</p>
		</div>
	</div>
</div>
	<div class="chapter" title="Chapter&nbsp;3.&nbsp;Tokenizer"><div class="titlepage"><div><div><h2 class="title"><a name="tools.tokenizer"></a>Chapter&nbsp;3.&nbsp;Tokenizer</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#tools.tokenizer.introduction">Tokenization</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.tokenizer.cmdline">Tokenizer Tools</a></span></dt><dt><span class="section"><a href="#tools.tokenizer.api">Tokenizer API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.tokenizer.training">Tokenizer Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.tokenizer.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.tokenizer.training.api">Training API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.tokenizer.detokenizing">Detokenizing</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.tokenizer.detokenizing.api">Detokenizing API</a></span></dt><dt><span class="section"><a href="#tools.tokenizer.detokenizing.dict">Detokenizer Dictionary</a></span></dt></dl></dd></dl></div>

	

	<div class="section" title="Tokenization"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.tokenizer.introduction"></a>Tokenization</h2></div></div></div>
		
		<p>
			The OpenNLP Tokenizers segment an input character sequence into
			tokens. Tokens are usually
			words, punctuation, numbers, etc.

			</p><pre class="programlisting">
			
Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29.
Mr. Vinken is chairman of Elsevier N.V., the Dutch publishing group.
Rudolph Agnew, 55 years old and former chairman of Consolidated Gold Fields
    PLC, was named a director of this British industrial conglomerate.
			
		 </pre><p>

			The following result shows the individual tokens in a whitespace
			separated representation.

			</p><pre class="programlisting">
			
Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields PLC ,
    was named a nonexecutive director of this British industrial conglomerate . 
A form of asbestos once used to make Kent cigarette filters has caused a high
    percentage of cancer deaths among a group of workers exposed to it more than 30 years ago ,
    researchers reported . 
			
		 	</pre><p>

			OpenNLP offers multiple tokenizer implementations:
			</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
					<p>Whitespace Tokenizer - A whitespace tokenizer, non whitespace
						sequences are identified as tokens</p>
				</li><li class="listitem">
					<p>Simple Tokenizer - A character class tokenizer, sequences of
						the same character class are tokens</p>
				</li><li class="listitem">
					<p>Learnable Tokenizer - A maximum entropy tokenizer, detects
						token boundaries based on probability model</p>
				</li></ul></div><p>

			Most part-of-speech taggers, parsers and so on, work with text
			tokenized in this manner. It is important to ensure that your
			tokenizer
			produces tokens of the type expected by your later text
			processing
			components.
		</p>

		<p>
			With OpenNLP (as with many systems), tokenization is a two-stage
			process:
			first, sentence boundaries are identified, then tokens within
			each
			sentence are identified.
		</p>
	
	<div class="section" title="Tokenizer Tools"><div class="titlepage"><div><div><h3 class="title"><a name="tools.tokenizer.cmdline"></a>Tokenizer Tools</h3></div></div></div>
		
		<p>The easiest way to try out the tokenizers are the command line
			tools. The tools are only intended for demonstration and testing.
		</p>
		<p>There are two tools, one for the Simple Tokenizer and one for
			the learnable tokenizer. A command line tool the for the Whitespace
			Tokenizer does not exist, because the whitespace separated output
			would be identical to the input.</p>
		<p>
			The following command shows how to use the Simple Tokenizer Tool.

			</p><pre class="programlisting">
			
$ bin/opennlp SimpleTokenizer
			
		 </pre><p>
			To use the learnable tokenizer download the english token model from
			our website.
			</p><pre class="programlisting">
			
$ bin/opennlp TokenizerME en-token.bin
			
		 </pre><p>
			To test the tokenizer copy the sample from above to the console. The
			whitespace separated tokens will be written written back to the
			console.
		</p>
		<p>
			Usually the input is read from a file and written to a file.
			</p><pre class="programlisting">
			
$ bin/opennlp TokenizerME en-token.bin &lt; article.txt &gt; article-tokenized.txt
			
		 </pre><p>
			It can be done in the same way for the Simple Tokenizer.
		</p>
		<p>
			Since most text comes truly raw and doesn't have sentence boundaries
			and such, its possible to create a pipe which first performs sentence
			boundary detection and tokenization. The following sample illustrates
			that.
			</p><pre class="screen">
			
$ opennlp SentenceDetector sentdetect.model &lt; article.txt | opennlp TokenizerME tokenize.model | more
Loading model ... Loading model ... done
done
Showa Shell gained 20 to 1,570 and Mitsubishi Oil rose 50 to 1,500.
Sumitomo Metal Mining fell five yen to 692 and Nippon Mining added 15 to 960 .
Among other winners Wednesday was Nippon Shokubai , which was up 80 at 2,410 .
Marubeni advanced 11 to 890 .
London share prices were bolstered largely by continued gains on Wall Street and technical 
    factors affecting demand for London 's blue-chip stocks .
...etc...
		 </pre><p>
			Of course this is all on the command line. Many people use the models
			directly in their Java code by creating SentenceDetector and
			Tokenizer objects and calling their methods as appropriate. The
			following section will explain how the Tokenizers can be used
			directly from java.
		</p>
	</div>

	<div class="section" title="Tokenizer API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.tokenizer.api"></a>Tokenizer API</h3></div></div></div>
		
		<p>
			The Tokenizers can be integrated into an application by the defined
			API.
			The shared instance of the WhitespaceTokenizer can be retrieved from a
			static field WhitespaceTokenizer.INSTANCE. The shared instance of the
			SimpleTokenizer can be retrieved in the same way from
			SimpleTokenizer.INSTANCE.
			To instantiate the TokenizerME (the learnable tokenizer) a Token Model
			must be created first. The following code sample shows how a model
			can be loaded.
			</p><pre class="programlisting">
			
InputStream modelIn = <b class="hl-keyword">new</b> FileInputStream(<b class="hl-string"><i style="color:red">"en-token.bin"</i></b>);

<b class="hl-keyword">try</b> {
  TokenizerModel model = <b class="hl-keyword">new</b> TokenizerModel(modelIn);
}
<b class="hl-keyword">catch</b> (IOException e) {
  e.printStackTrace();
}
<b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (modelIn != null) {
    <b class="hl-keyword">try</b> {
      modelIn.close();
    }
    <b class="hl-keyword">catch</b> (IOException e) {
    }
  }
}
			
		 </pre><p>
			After the model is loaded the TokenizerME can be instantiated.
			</p><pre class="programlisting">
			
Tokenizer tokenizer = <b class="hl-keyword">new</b> TokenizerME(model);
		 </pre><p>
			The tokenizer offers two tokenize methods, both expect an input
			String object which contains the untokenized text. If possible it
			should be a sentence, but depending on the training of the learnable
			tokenizer this is not required. The first returns an array of
			Strings, where each String is one token.
			</p><pre class="programlisting">
			
String tokens[] = tokenizer.tokenize(<b class="hl-string"><i style="color:red">"An input sample sentence."</i></b>);
		 </pre><p>
			The output will be an array with these tokens.
			</p><pre class="programlisting">
			
"An", "input", "sample", "sentence", "."
		 </pre><p>
			The second method, tokenizePos returns an array of Spans, each Span
			contain the begin and end character offsets of the token in the input
			String.
			</p><pre class="programlisting">
			
Span tokenSpans[] = tokenizer.tokenizePos(<b class="hl-string"><i style="color:red">"An input sample sentence."</i></b>);		
			</pre><p>
			The tokenSpans array now contain 5 elements. To get the text for one
			span call Span.getCoveredText which takes a span and the input text.

			The TokenizerME is able to output the probabilities for the detected
			tokens. The getTokenProbabilities method must be called directly
			after one of the tokenize methods was called.
			</p><pre class="programlisting">
			
TokenizerME tokenizer = ...

String tokens[] = tokenizer.tokenize(...);
<b class="hl-keyword">double</b> tokenProbs[] = tokenizer.getTokenProbabilities();
					
			</pre><p>
			The tokenProbs array now contains one double value per token, the
			value is between 0 and 1, where 1 is the highest possible probability
			and 0 the lowest possible probability.
		</p>
	</div>
	</div>
	
	<div class="section" title="Tokenizer Training"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.tokenizer.training"></a>Tokenizer Training</h2></div></div></div>
		
			
		<div class="section" title="Training Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.tokenizer.training.tool"></a>Training Tool</h3></div></div></div>
			
			<p>
				OpenNLP has a command line tool which is used to train the models
				available from the model download page on various corpora. The data
				must be converted to the OpenNLP Tokenizer training format. Which is
				one sentence per line. Tokens are either separater by a whitespace or
				if by a special &lt;SPLIT&gt; tag.
				
				The following sample shows the sample from above in the correct format.
				</p><pre class="programlisting">
			
Pierre Vinken&lt;SPLIT&gt;, 61 years old&lt;SPLIT&gt;, will join the board as a nonexecutive director Nov. 29&lt;SPLIT&gt;.
Mr. Vinken is chairman of Elsevier N.V.&lt;SPLIT&gt;, the Dutch publishing group&lt;SPLIT&gt;.
Rudolph Agnew&lt;SPLIT&gt;, 55 years old and former chairman of Consolidated Gold Fields PLC&lt;SPLIT&gt;,
    was named a nonexecutive director of this British industrial conglomerate&lt;SPLIT&gt;. 
					
			</pre><p>
			Usage of the tool:
			</p><pre class="screen">
			
$ bin/opennlp TokenizerTrainer
Usage: opennlp TokenizerTrainer [-abbDict path] [-alphaNumOpt isAlphaNumOpt] [-params paramsFile] -lang language [-cutoff num] [-iterations num] [-encoding charsetName] -data trainData -model modelFile

Arguments description:
	-abbDict path
		The abbreviation dictionary in XML format.
	-alphaNumOpt isAlphaNumOpt
		Optimization flag to skip alpha numeric tokens for further tokenization
	-params paramsFile
		Training parameters file.
	-lang language
		specifies the language which is being processed.
	-cutoff num
		specifies the min number of times a feature must be seen. It is ignored if a parameters file is passed.
	-iterations num
		specifies the number of training iterations. It is ignored if a parameters file is passed.
	-encoding charsetName
		specifies the encoding which should be used for reading and writing text. If not specified the system default will be used.
	-data trainData
		the data to be used during training
	-model modelFile
		the output model file
					
				</pre><p>
				To train the english tokenizer use the following command:
				</p><pre class="screen">
			
$ bin/opennlp TokenizerTrainer -encoding UTF-8 -lang en -alphaNumOpt \ 
+-data en-token.train -model en-token.bin
Indexing events using cutoff of 5

	Computing event counts...  done. 262271 events
	Indexing...  done.
Sorting and merging events... done. Reduced 262271 events to 59060.
Done indexing.
Incorporating indexed data for training...  
done.
	Number of Event Tokens: 59060
	    Number of Outcomes: 2
	  Number of Predicates: 15695
...done.
Computing model parameters...
Performing 100 iterations.
  1:  .. loglikelihood=-181792.40419263614	0.9614292087192255
  2:  .. loglikelihood=-34208.094253153664	0.9629238459456059
  3:  .. loglikelihood=-18784.123872910015	0.9729211388220581
  4:  .. loglikelihood=-13246.88162585859	0.9856103038460219
  5:  .. loglikelihood=-10209.262670265718	0.9894422181636552

 ...&lt;skipping a bunch of iterations&gt;...

 95:  .. loglikelihood=-769.2107474529454	0.999511955191386
 96:  .. loglikelihood=-763.8891914534009	0.999511955191386
 97:  .. loglikelihood=-758.6685383254891	0.9995157680414533
 98:  .. loglikelihood=-753.5458314695236	0.9995157680414533
 99:  .. loglikelihood=-748.5182305519613	0.9995157680414533
100:  .. loglikelihood=-743.5830058068038	0.9995157680414533
Wrote tokenizer model.
Path: en-token.bin
					
				</pre><p>
			</p>
		</div>
		<div class="section" title="Training API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.tokenizer.training.api"></a>Training API</h3></div></div></div>
			
			<p>TODO: Write documentation about the tokenizer training api. Any contributions
are very welcome. If you want to contribute please contact us on the mailing list
or comment on the jira issue <a class="ulink" href="https://issues.apache.org/jira/browse/OPENNLP-215" target="_top">OPENNLP-215</a>.</p>
		</div>
	</div>
	
	<div class="section" title="Detokenizing"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.tokenizer.detokenizing"></a>Detokenizing</h2></div></div></div>
		
		<p>
		Detokenizing is simple the opposite of tokenization, the original non-tokenized string should
		be constructed out of a token sequence. The OpenNLP implementation was created to undo the tokenization
		of training data for the tokenizer. It can also be used to undo the tokenization of such a trained
		tokenizer. The implementation is strictly rule based and defines how tokens should be attached
		to a sentence wise character sequence.
		</p>
		<p>
		The rule dictionary assign to every token an operation which describes how it should be attached
		to one continous character sequence.
		</p>
		<p>
		The following rules can be assigned to a token:
		</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
				<p>MERGE_TO_LEFT - Merges the token to the left side.</p>
			</li><li class="listitem">
				<p>MERGE_TO_RIGHT - Merges the token to the righ side.</p>
			</li><li class="listitem">
				<p>RIGHT_LEFT_MATCHING - Merges the token to the right side on first occurence
				and to the left side on second occurence.</p>
			</li></ul></div><p>
		
		The following sample will illustrate how the detokenizer with a small
		rule dictionary (illustration format, not the xml data format):
		</p><pre class="programlisting">
			
. MERGE_TO_LEFT
" RIGHT_LEFT_MATCHING		
		</pre><p>
		The dictionary should be used to de-tokenize the following whitespace tokenized sentence:
		</p><pre class="programlisting">
			
He said " This is a test " .		
		</pre><p>	
		The tokens would get these tags based on the dictionary:
		</p><pre class="programlisting">
			
He -&gt; NO_OPERATION
said -&gt; NO_OPERATION
" -&gt; MERGE_TO_RIGHT
This -&gt; NO_OPERATION
is -&gt; NO_OPERATION
a -&gt; NO_OPERATION
test -&gt; NO_OPERATION
" -&gt; MERGE_TO_LEFT
. -&gt; MERGE_TO_LEFT		
			</pre><p>
			That will result in the following character sequence:
		</p><pre class="programlisting">
			
He said "This is a test".		
		</pre><p>
		TODO: Add documentation about the dictionary format and how to use the API. Contributions are welcome.	
		</p>
		<div class="section" title="Detokenizing API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.tokenizer.detokenizing.api"></a>Detokenizing API</h3></div></div></div>
			
			<p>TODO: Write documentation about the detokenizer api. Any contributions
are very welcome. If you want to contribute please contact us on the mailing list
or comment on the jira issue <a class="ulink" href="https://issues.apache.org/jira/browse/OPENNLP-216" target="_top">OPENNLP-216</a>.</p>
		</div>
		<div class="section" title="Detokenizer Dictionary"><div class="titlepage"><div><div><h3 class="title"><a name="tools.tokenizer.detokenizing.dict"></a>Detokenizer Dictionary</h3></div></div></div>
			
			<p>TODO: Write documentation about the detokenizer dictionary. Any contributions
are very welcome. If you want to contribute please contact us on the mailing list
or comment on the jira issue <a class="ulink" href="https://issues.apache.org/jira/browse/OPENNLP-217" target="_top">OPENNLP-217</a>.</p>
		</div>
	</div>
</div>
	<div class="chapter" title="Chapter&nbsp;4.&nbsp;Name Finder"><div class="titlepage"><div><div><h2 class="title"><a name="tools.namefind"></a>Chapter&nbsp;4.&nbsp;Name Finder</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#tools.namefind.recognition">Named Entity Recognition</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.namefind.recognition.cmdline">Name Finder Tool</a></span></dt><dt><span class="section"><a href="#tools.namefind.recognition.api">Name Finder API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.namefind.training">Name Finder Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.namefind.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.namefind.training.api">Training API</a></span></dt><dt><span class="section"><a href="#tools.namefind.training.featuregen">Custom Feature Generation</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.namefind.eval">Evaluation</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.namefind.eval.tool">Evaluation Tool</a></span></dt><dt><span class="section"><a href="#tools.namefind.eval.api">Evaluation API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.namefind.annotation_guides">Named Entity Annotation Guidelines</a></span></dt></dl></div>

	

	<div class="section" title="Named Entity Recognition"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.namefind.recognition"></a>Named Entity Recognition</h2></div></div></div>
		
		<p>
			The Name Finder can detect named entities and numbers in text. To be able to
			detect entities the Name Finder needs a model. The model is dependent on the
			language and entity type it was trained for. The OpenNLP projects offers a number
			of pre-trained name finder models which are trained on various freely available corpora.
			They can be downloaded at our model download page. To find names in raw text the text
			must be segmented into tokens and sentences. A detailed description is given in the
			sentence detector and tokenizer tutorial. Its important that the tokenization for
			the training data and the input text is identical.
		</p>
	
	<div class="section" title="Name Finder Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.namefind.recognition.cmdline"></a>Name Finder Tool</h3></div></div></div>
		
		<p>
			The easiest way to try out the Name Finder is the command line tool.
			The tool is only intended for demonstration and testing. Download the
			English
			person model and start the Name Finder Tool with this command:
			</p><pre class="screen">
				
$bin/opennlp TokenNameFinder en-ner-person.bin
			 </pre><p>
			 
			The name finder now reads a tokenized sentence per line from stdin, an empty
			line indicates a document boundary and resets the adaptive feature generators.
			Just copy this text to the terminal:
	
			</p><pre class="programlisting">
				
Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr . Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields PLC , was named
    a director of this British industrial conglomerate .
			 </pre><p>
			 the name finder will now output the text with markup for person names:
			</p><pre class="programlisting">
				
&lt;START:person&gt; Pierre Vinken &lt;END&gt; , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr . &lt;START:person&gt; Vinken &lt;END&gt; is chairman of Elsevier N.V. , the Dutch publishing group .
&lt;START:person&gt; Rudolph Agnew &lt;END&gt; , 55 years old and former chairman of Consolidated Gold Fields PLC ,
    was named a director of this British industrial conglomerate .
				
			 </pre><p>		 
		</p>
	</div>
		<div class="section" title="Name Finder API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.namefind.recognition.api"></a>Name Finder API</h3></div></div></div>
		
		<p>
			To use the Name Finder in a production system its strongly recommended to embed it
			directly into the application instead of using the command line interface.
			First the name finder model must be loaded into memory from disk or an other source.
			In the sample below its loaded from disk.
			</p><pre class="programlisting">
				
InputStream modelIn = <b class="hl-keyword">new</b> FileInputStream(<b class="hl-string"><i style="color:red">"en-ner-person.bin"</i></b>);

<b class="hl-keyword">try</b> {
  TokenNameFinderModel model = <b class="hl-keyword">new</b> TokenNameFinderModel(modelIn);
}
<b class="hl-keyword">catch</b> (IOException e) {
  e.printStackTrace();
}
<b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (modelIn != null) {
    <b class="hl-keyword">try</b> {
      modelIn.close();
    }
    <b class="hl-keyword">catch</b> (IOException e) {
    }
  }
}
			 </pre><p>
			 There is a number of reasons why the model loading can fail:
			 </p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
					<p>Issues with the underlying I/O</p>
				</li><li class="listitem">
					<p>The version of the model is not compatible with the OpenNLP version</p>
				</li><li class="listitem">
					<p>The model is loaded into the wrong component,
					for example a tokenizer model is loaded with TokenNameFinderModel class.</p>
				</li><li class="listitem">
					<p>The model content is not valid for some other reason</p>
				</li></ul></div><p>
			After the model is loaded the NameFinderME can be instantiated.
			</p><pre class="programlisting">
				
NameFinderME nameFinder = <b class="hl-keyword">new</b> NameFinderME(model);
			</pre><p>
			The initialization is now finished and the Name Finder can be used. The NameFinderME
			class is not thread safe, it must only be called from one thread. To use multiple threads
			multiple NameFinderME instances sharing the same model instance can be created.
			The input text should be segmented into documents, sentences and tokens.
			To perform entity detection an application calls the find method for every sentence in the
			document. After every document clearAdaptiveData must be called to clear the adaptive data in
			the feature generators. Not calling clearAdaptiveData can lead to a sharp drop in the detection
			rate after a few documents.
			The following code illustrates that:
			</p><pre class="programlisting">
				
<b class="hl-keyword">for</b> (String document[][] : documents) {

  <b class="hl-keyword">for</b> (String[] sentence : document) {
    Span nameSpans[] = nameFinder.find(sentence);
    <i class="hl-comment" style="color: silver">// do something with the names</i>
  }

  nameFinder.clearAdaptiveData()
}
			 </pre><p>
			 the following snippet shows a call to find
			 </p><pre class="programlisting">
				
String sentence[] = <b class="hl-keyword">new</b> String[]{
    <b class="hl-string"><i style="color:red">"Pierre"</i></b>,
    <b class="hl-string"><i style="color:red">"Vinken"</i></b>,
    <b class="hl-string"><i style="color:red">"is"</i></b>,
    <b class="hl-string"><i style="color:red">"61"</i></b>,
    <b class="hl-string"><i style="color:red">"years"</i></b>
    <b class="hl-string"><i style="color:red">"old"</i></b>,
    <b class="hl-string"><i style="color:red">"."</i></b>
    };

Span nameSpans[] = nameFinder.find(sentence);
			</pre><p>
			The nameSpans arrays contains now exactly one Span which marks the name Pierre Vinken. 
			The elements between the begin and end offsets are the name tokens. In this case the begin 
			offset is 0 and the end offset is 2. The Span object also knows the type of the entity.
			In this case its person (defined by the model). It can be retrieved with a call to Span.getType().
			Additionally to the statistical Name Finder, OpenNLP also offers a dictionary and a regular
			expression name finder implementation.
		</p>
		<p>
			TODO: Explain how to retrieve probs from the name finder for names and for non recognized names
		</p>
	</div>
	</div>
	<div class="section" title="Name Finder Training"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.namefind.training"></a>Name Finder Training</h2></div></div></div>
		
		<p>
			The pre-trained models might not be available for a desired language, can not detect
			important entities or the performance is not good enough outside the news domain.
			These are the typical reason to do custom training of the name finder on a new corpus
			or on a corpus which is extended by private training data taken from the data which should be analyzed.
		</p>
		
		<div class="section" title="Training Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.namefind.training.tool"></a>Training Tool</h3></div></div></div>
		
		<p>
			OpenNLP has a command line tool which is used to train the models available from the model
			download page on various corpora.
		</p>
		<p>
			The data must be converted to the OpenNLP name finder training format. Which is one sentence per line.
			The sentence must be tokenized and contain spans which mark the entities. Documents are separated by
			empty lines which trigger the reset of the adaptive feature generators. A training file can contain
			multiple types. If the training file contains multiple types the created model will also be able to
			detect these multiple types. For now its recommended to only train single type models, since multi
			type support is stil experimental.
		</p>
		<p>
			Sample sentence of the data:
			</p><pre class="programlisting">
				
&lt;START:person&gt; Pierre Vinken &lt;END&gt; , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr . &lt;START:person&gt; Vinken &lt;END&gt; is chairman of Elsevier N.V. , the Dutch publishing group .
				
			 </pre><p>
			 The training data should contain at least 15000 sentences to create a model which performs well.
			 Usage of the tool:
			</p><pre class="screen">
				
$ bin/opennlp TokenNameFinderTrainer
Usage: opennlp TokenNameFinderTrainer -lang language -encoding charset [-iterations num] \ 
[-cutoff num] [-type type] -data trainingData -model model
-lang language     specifies the language which is being processed.
-encoding charset  specifies the encoding which should be used for reading and writing text.
-iterations num    specified the number of training iterations
-cutoff num        specifies the min number of times a feature must be seen
-type The type of the token name finder model
			 </pre><p>
			 Its now assumed that the english person name finder model should be trained from a file
			 called en-ner-person.train which is encoded as UTF-8. The following command will train
			 the name finder and write the model to en-ner-person.bin:
			 </p><pre class="screen">
				
$bin/opennlp TokenNameFinderTrainer -encoding UTF-8 -lang en -data en-ner-person.train -model en-ner-person.bin
			 </pre><p>
			 Additionally its possible to specify the number of iterations,
			 the cutoff and to overwrite all types in the training data with a single type.
		</p>
		</div>
		<div class="section" title="Training API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.namefind.training.api"></a>Training API</h3></div></div></div>
		
		<p>
			To train the name finder from within an application its recommended to use the training
			API instead of the command line tool.
			Basically three steps are necessary to train it:
			</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
					<p>The application must open a sample data stream</p>
				</li><li class="listitem">
					<p>Call the NameFinderME.train method</p>
				</li><li class="listitem">
					<p>Save the TokenNameFinderModel to a file or database</p>
				</li></ul></div><p>
			The three steps are illustrated by the following sample code:
			</p><pre class="programlisting">
				
Charset charset = Charset.forName(<b class="hl-string"><i style="color:red">"UTF-8"</i></b>);
ObjectStream&lt;String&gt; lineStream =
		<b class="hl-keyword">new</b> PlainTextByLineStream(<b class="hl-keyword">new</b> FileInputStream(<b class="hl-string"><i style="color:red">"en-ner-person.train"</i></b>), charset);
ObjectStream&lt;NameSample&gt; sampleStream = <b class="hl-keyword">new</b> NameSampleDataStream(lineStream);

TokenNameFinderModel model;

<b class="hl-keyword">try</b> {
  model = NameFinderME.train(<b class="hl-string"><i style="color:red">"en"</i></b>, <b class="hl-string"><i style="color:red">"person"</i></b>, sampleStream,
      Collections.&lt;String, Object&gt;emptyMap(), <span class="hl-number">100</span>, <span class="hl-number">5</span>);
}
<b class="hl-keyword">finally</b> {
  sampleStream.close();
}

<b class="hl-keyword">try</b> {
  modelOut = <b class="hl-keyword">new</b> BufferedOutputStream(<b class="hl-keyword">new</b> FileOutputStream(modelFile));
  model.serialize(modelOut);
} <b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (modelOut != null) 
     modelOut.close();      
}
			 </pre><p>
		</p>
		</div>
		
		<div class="section" title="Custom Feature Generation"><div class="titlepage"><div><div><h3 class="title"><a name="tools.namefind.training.featuregen"></a>Custom Feature Generation</h3></div></div></div>
		
			<p>
				OpenNLP defines a default feature generation which is used when no custom feature
				generation is specified. Users which want to experiment with the feature generation
				can provide a custom feature generator. Either via API or via an xml descriptor file.
			</p>
			<div class="section" title="Feature Generation defined by API"><div class="titlepage"><div><div><h4 class="title"><a name="tools.namefind.training.featuregen.api"></a>Feature Generation defined by API</h4></div></div></div>
			
			<p>
				The custom generator must be used for training
				and for detecting the names. If the feature generation during training time and detection
				time is different the name finder might not be able to detect names.
				The following lines show how to construct a custom feature generator
				</p><pre class="programlisting">
					
	AdaptiveFeatureGenerator featureGenerator = <b class="hl-keyword">new</b> CachedFeatureGenerator(
	         <b class="hl-keyword">new</b> AdaptiveFeatureGenerator[]{
	           <b class="hl-keyword">new</b> WindowFeatureGenerator(<b class="hl-keyword">new</b> TokenFeatureGenerator(), <span class="hl-number">2</span>, <span class="hl-number">2</span>),
	           <b class="hl-keyword">new</b> WindowFeatureGenerator(<b class="hl-keyword">new</b> TokenClassFeatureGenerator(true), <span class="hl-number">2</span>, <span class="hl-number">2</span>),
	           <b class="hl-keyword">new</b> OutcomePriorFeatureGenerator(),
	           <b class="hl-keyword">new</b> PreviousMapFeatureGenerator(),
	           <b class="hl-keyword">new</b> BigramNameFeatureGenerator(),
	           <b class="hl-keyword">new</b> SentenceFeatureGenerator(true, false)
	           });
				</pre><p>
				which is similar to the default feature generator.
				The javadoc of the feature generator classes explain what the individual feature generators do.
				To write a custom feature generator please implement the AdaptiveFeatureGenerator interface or
				if it must not be adaptive extend the FeatureGeneratorAdapter.
				The train method which should be used is defined as
				</p><pre class="programlisting">
					
<b class="hl-keyword">public</b> <b class="hl-keyword">static</b> TokenNameFinderModel train(String languageCode, String type, ObjectStream&lt;NameSample&gt; samples, 
       AdaptiveFeatureGenerator generator, <b class="hl-keyword">final</b> Map&lt;String, Object&gt; resources, 
       <b class="hl-keyword">int</b> iterations, <b class="hl-keyword">int</b> cutoff) <b class="hl-keyword">throws</b> IOException
				</pre><p>
				and can take feature generator as an argument.
				To detect names the model which was returned from the train method and the
				feature generator must be passed to the NameFinderME constructor.
				</p><pre class="programlisting">
					
<b class="hl-keyword">new</b> NameFinderME(model, featureGenerator, NameFinderME.DEFAULT_BEAM_SIZE);
				 </pre><p>	 
			</p>
			</div>
			<div class="section" title="Feature Generation defined by XML Descriptor"><div class="titlepage"><div><div><h4 class="title"><a name="tools.namefind.training.featuregen.xml"></a>Feature Generation defined by XML Descriptor</h4></div></div></div>
			
			<p>
			OpenNLP can also use a xml descritpor file to configure the featuer generation. The descriptor
			file is stored inside the model after training and the feature generators are configured
			correctly when the name finder is instantiated.
			
			The following sample shows a xml descriptor:
				</p><pre class="programlisting">
					
<b class="hl-tag" style="color: #000096">&lt;generators&gt;</b>
  <b class="hl-tag" style="color: #000096">&lt;cache&gt;</b> 
    <b class="hl-tag" style="color: #000096">&lt;generators&gt;</b>
      <b class="hl-tag" style="color: #000096">&lt;window</b> <span class="hl-attribute" style="color: #F5844C">prevLength</span> = <span class="hl-value" style="color: #993300">"2"</span> <span class="hl-attribute" style="color: #F5844C">nextLength</span> = <span class="hl-value" style="color: #993300">"2"</span><b class="hl-tag" style="color: #000096">&gt;</b>          
        <b class="hl-tag" style="color: #000096">&lt;tokenclass/&gt;</b>
      <b class="hl-tag" style="color: #000096">&lt;/window&gt;</b>
      <b class="hl-tag" style="color: #000096">&lt;window</b> <span class="hl-attribute" style="color: #F5844C">prevLength</span> = <span class="hl-value" style="color: #993300">"2"</span> <span class="hl-attribute" style="color: #F5844C">nextLength</span> = <span class="hl-value" style="color: #993300">"2"</span><b class="hl-tag" style="color: #000096">&gt;</b>                
        <b class="hl-tag" style="color: #000096">&lt;token/&gt;</b>
      <b class="hl-tag" style="color: #000096">&lt;/window&gt;</b>
      <b class="hl-tag" style="color: #000096">&lt;definition/&gt;</b>
      <b class="hl-tag" style="color: #000096">&lt;prevmap/&gt;</b>
      <b class="hl-tag" style="color: #000096">&lt;bigram/&gt;</b>
      <b class="hl-tag" style="color: #000096">&lt;sentence</b> <span class="hl-attribute" style="color: #F5844C">begin</span>=<span class="hl-value" style="color: #993300">"true"</span> <span class="hl-attribute" style="color: #F5844C">end</span>=<span class="hl-value" style="color: #993300">"false"</span><b class="hl-tag" style="color: #000096">/&gt;</b>
    <b class="hl-tag" style="color: #000096">&lt;/generators&gt;</b>
  <b class="hl-tag" style="color: #000096">&lt;/cache&gt;</b> 
<b class="hl-tag" style="color: #000096">&lt;/generators&gt;</b>
				 </pre><p>
		    The root element must be generators, each sub-element adds a feature generator to the configuration.
		    The sample xml is equivalent to the generators defined by the API above.
			</p>
			<p>
			The following table shows the supported elements:
			</p><div class="table"><a name="d4e205"></a><p class="title"><b>Table&nbsp;4.1.&nbsp;Genertor elements</b></p><div class="table-contents">
			  
			  <table summary="Genertor elements" border="1"><colgroup><col><col></colgroup><thead><tr><th>Element</th><th>Aggregated</th><th>Attributes</th></tr></thead><tbody><tr><td>generators</td><td>yes</td><td>none</td></tr><tr><td>cache</td><td>yes</td><td>none</td></tr><tr><td>charngram</td><td>no</td><td><span class="emphasis"><em>min</em></span> and <span class="emphasis"><em>max</em></span> specify the length of the generated character ngrams</td></tr><tr><td>definition</td><td>no</td><td>none</td></tr><tr><td>dictionary</td><td>no</td><td><span class="emphasis"><em>dict</em></span> is the key of the dictionary resource to use,
					       and <span class="emphasis"><em>prefix</em></span> is a feature prefix string</td></tr><tr><td>prevmap</td><td>no</td><td>none</td></tr><tr><td>sentence</td><td>no</td><td><span class="emphasis"><em>begin</em></span> and <span class="emphasis"><em>end</em></span> to generate begin or end features, both are optional and are boolean values</td></tr><tr><td>tokenclass</td><td>no</td><td>none</td></tr><tr><td>token</td><td>no</td><td>none</td></tr><tr><td>bigram</td><td>no</td><td>none</td></tr><tr><td>tokenpattern</td><td>no</td><td>none</td></tr><tr><td>window</td><td>yes</td><td><span class="emphasis"><em>prevLength</em></span> and <span class="emphasis"><em>nextLength</em></span> must be integers ans specify the window size</td></tr><tr><td>custom</td><td>no</td><td><span class="emphasis"><em>class</em></span> is the name of the feature generator class whcih will be loaded</td></tr></tbody></table>
			</div></div><p><br class="table-break">
			Aggregated feature generators can contain other generators, like the cache or the window feature
			generator in the sample.
			</p>
			</div>
		</div>
	</div>
	<div class="section" title="Evaluation"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.namefind.eval"></a>Evaluation</h2></div></div></div>
		
		<p>
		The built in evaluation can measure the named entity recognition performance of the name finder.
		The performance is either measured on a test dataset or via cross validation.
		</p>
		<div class="section" title="Evaluation Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.namefind.eval.tool"></a>Evaluation Tool</h3></div></div></div>
		
		<p>
		The following command shows how the tool can be run:
		</p><pre class="screen">
				
$bin/opennlp TokenNameFinderEvaluator -encoding UTF-8 -model en-ner-person.bin -data en-ner-person.test
			 </pre><p>
			 </p><pre class="screen">
				
Precision: 0.8005071889818507
Recall: 0.7450581122145297
F-Measure: 0.7717879983140168
			 </pre><p>
			 Note: The command line interface does not support cross evaluation in the current version. 
		</p> 
		</div>
		<div class="section" title="Evaluation API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.namefind.eval.api"></a>Evaluation API</h3></div></div></div>
		
		<p>
		The evaluation can be performed on a pre-trained model and a test dataset or via cross validation.
		In the first case the model must be loaded and a NameSample ObjectStream must be created (see code samples above),
		assuming these two objects exist the following code shows how to perform the evaluation:
			</p><pre class="programlisting">
				
TokenNameFinderEvaluator evaluator = <b class="hl-keyword">new</b> TokenNameFinderEvaluator(<b class="hl-keyword">new</b> NameFinderME(model));
evaluator.evaluate(sampleStream);

FMeasure result = evaluator.getFMeasure();

System.out.println(result.toString());
			</pre><p>
			In the cross validation case all the training arguments must be
			provided (see the Training API section above).
			To perform cross validation the ObjectStream must be resettable.
			</p><pre class="programlisting">
				
FileInputStream sampleDataIn = <b class="hl-keyword">new</b> FileInputStream(<b class="hl-string"><i style="color:red">"en-ner-person.train"</i></b>);
ObjectStream&lt;NameSample&gt; sampleStream = <b class="hl-keyword">new</b> PlainTextByLineStream(sampleDataIn.getChannel(), <b class="hl-string"><i style="color:red">"UTF-8"</i></b>); 
TokenNameFinderCrossValidator evaluator = <b class="hl-keyword">new</b> TokenNameFinderCrossValidator(<b class="hl-string"><i style="color:red">"en"</i></b>, <span class="hl-number">100</span>, <span class="hl-number">5</span>);
evaluator.evaluate(sampleStream, <span class="hl-number">10</span>);

FMeasure result = evaluator.getFMeasure();

System.out.println(result.toString());
			</pre><p>
		</p>
		</div>
	</div>
			<div class="section" title="Named Entity Annotation Guidelines"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.namefind.annotation_guides"></a>Named Entity Annotation Guidelines</h2></div></div></div>
		
		<p>
			Annotation guidelines define what should be labeled as an entity. To build
			a private corpus its important to know these guidelines and maybe write a
			custom one.
			Here is a list of publicly available annotation guidelines:
				</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
					<p>
						<a class="ulink" href="http://cs.nyu.edu/cs/faculty/grishman/NEtask20.book_1.html" target="_top">
							MUC6
						</a>
					</p>
				</li><li class="listitem">
					<p>
						<a class="ulink" href="http://acl.ldc.upenn.edu/muc7/ne_task.html" target="_top">
							MUC7
						</a>
					</p>
				</li><li class="listitem">
					<p>
						<a class="ulink" href="http://projects.ldc.upenn.edu/ace/docs/English-Entities-Guidelines_v5.6.1.pdf" target="_top">
							ACE
						</a>
					</p>
				</li><li class="listitem">
					<p>
						<a class="ulink" href="http://www.cnts.ua.ac.be/conll2003/ner/annotation.txt" target="_top">
							CONLL 2003
						</a>
					</p>
				</li></ul></div><p>
		</p>
		</div>
</div>
	<div class="chapter" title="Chapter&nbsp;5.&nbsp;Document Categorizer"><div class="titlepage"><div><div><h2 class="title"><a name="tools.doccat"></a>Chapter&nbsp;5.&nbsp;Document Categorizer</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#tools.doccat.classifying">Classifying</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.doccat.classifying.cmdline">Document Categorizer Tool</a></span></dt><dt><span class="section"><a href="#tools.doccat.classifying.api">Document Categorizer API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.doccat.training">Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.doccat.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.doccat.training.api">Training API</a></span></dt></dl></dd></dl></div>

	<div class="section" title="Classifying"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.doccat.classifying"></a>Classifying</h2></div></div></div>
		
		<p>
		The OpenNLP Document Categorizer can classify text into pre-defined categories. 
		It is based on maximum entropy framework. For someone interested in Gross Margin,
		the sample text given below could be classified as GMDecrease
				</p><pre class="programlisting">
			
Major acquisitions that have a lower gross margin than the existing network
also had a negative impact on the overall gross margin, but it should improve
following the implementation of its integration strategies.
		 </pre><p>
and the text below could be classified as GMIncrease
				</p><pre class="programlisting">
			
The upward movement of gross margin resulted from amounts pursuant to 
adjustments to obligations towards dealers.
		 </pre><p>
		 To be able to classify a text, the document categorizer needs a model. 
		 The classifications are requirements-specific
		 and hence there is no pre-built model for document categorizer under OpenNLP project.
		</p>		
	
	<div class="section" title="Document Categorizer Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.doccat.classifying.cmdline"></a>Document Categorizer Tool</h3></div></div></div>
		
		<p>
		The easiest way to try out the document categorizer is the command line tool. The tool is only
		intended for demonstration and testing. The following command shows how to use the document categorizer tool. 
		  </p><pre class="screen">
			
$ bin/opennlp Doccat model
		 </pre><p>
		 The input is read from standard input and output is written to standard output, unless they are redirected
		 or piped. As with most components in OpenNLP, document categorizer expects input which is segmented into sentences.
		</p>
 	 </div>
  	<div class="section" title="Document Categorizer API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.doccat.classifying.api"></a>Document Categorizer API</h3></div></div></div>
		
		<p>
			To perform classification you will need a maxent model -
			these are encapsulated in the DoccatModel class of OpenNLP tools.
		</p>
		<p>
			First you need to grab the bytes from the serialized model on an InputStream - 
			we'll leave it you to do that, since you were the one who serialized it to begin with. Now for the easy part:
						</p><pre class="programlisting">
				
InputStream is = ...
DoccatModel m = <b class="hl-keyword">new</b> DoccatModel(is);
				</pre><p>
				With the DoccatModel in hand we are just about there:
						</p><pre class="programlisting">
				
String inputText = ...
DocumentCategorizerME myCategorizer = <b class="hl-keyword">new</b> DocumentCategorierME(m);
<b class="hl-keyword">double</b>[] outcomes = myCategorizer.categorize(inputText);
String category = myCategorizer.getBestOutcome();
				</pre><p>
		</p>
	</div>
	</div>
	<div class="section" title="Training"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.doccat.training"></a>Training</h2></div></div></div>
	
		<p>
			The Document Categorizer can be trained on annotated training material. The data
			must be in OpenNLP Document Categorizer training format. This is one document per line,
			containing category and text separated by a whitespace.
			The following sample shows the sample from above in the required format. Here GMDecrease and GMIncrease
			are the categories.
			</p><pre class="programlisting">
			
GMDecrease Major acquisitions that have a lower gross margin than the existing network also \ 
           had a negative impact on the overall gross margin, but it should improve following \ 
           the implementation of its integration strategies .
GMIncrease The upward movement of gross margin resulted from amounts pursuant to adjustments \
           to obligations towards dealers .
			</pre><p>
			Note: The line breaks marked with a backslash are just inserted for formatting purposes and must not be
			included in the training data. 
		</p>
		<div class="section" title="Training Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.doccat.training.tool"></a>Training Tool</h3></div></div></div>
		
		<p>
		The following command will train the document categorizer and write the model to en-doccat.bin:		
		  </p><pre class="screen">
						
$bin/opennlp DoccatTrainer -encoding UTF-8 -lang en -data en-doccat.train -model en-doccat.bin
		 </pre><p>
		Additionally it is possible to specify the number of iterations, and the cutoff.
		</p>
		</div>
				<div class="section" title="Training API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.doccat.training.api"></a>Training API</h3></div></div></div>
		
		<p>
		So, naturally you will need some access to many pre-classified events to train your model.
		The class opennlp.tools.doccat.DocumentSample encapsulates a text document and its classification.
		DocumentSample has two constructors. Each take the text's category as one argument. The other argument can either be raw
		text, or an array of tokens. By default, the raw text will be split into tokens by whitespace. So, let's say
		your training data was contained in a text file, where the format is as described above.
		Then you might want to write something like this to create a collection of DocumentSamples:
		</p><pre class="programlisting">
						
DoccatModel model = null;

InputStream dataIn = null;
<b class="hl-keyword">try</b> {
  dataIn = <b class="hl-keyword">new</b> FileInputStream(<b class="hl-string"><i style="color:red">"en-sentiment.train"</i></b>);
  ObjectStream&lt;String&gt; lineStream =
		<b class="hl-keyword">new</b> PlainTextByLineStream(dataIn, <b class="hl-string"><i style="color:red">"UTF-8"</i></b>);
  ObjectStream&lt;DocumentSample&gt; sampleStream = <b class="hl-keyword">new</b> DocumentSampleStream(lineStream);

  model = DocumentCategorizerME.train(<b class="hl-string"><i style="color:red">"en"</i></b>, sampleStream);
}
<b class="hl-keyword">catch</b> (IOException e) {
  <i class="hl-comment" style="color: silver">// Failed to read or parse training data, training failed</i>
  e.printStackTrace();
}
<b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (dataIn != null) {
    <b class="hl-keyword">try</b> {
      dataIn.close();
    }
    <b class="hl-keyword">catch</b> (IOException e) {
      <i class="hl-comment" style="color: silver">// Not an issue, training already finished.</i>
      <i class="hl-comment" style="color: silver">// The exception should be logged and investigated</i>
      <i class="hl-comment" style="color: silver">// if part of a production system.</i>
      e.printStackTrace();
    }
  }
}
	</pre><p>
	Now might be a good time to cruise over to Hulu or something, because this could take a while if you've got a large training set.
	You may see a lot of output as well. Once you're done, you can pretty quickly step to classification directly,
	but first we'll cover serialization. Feel free to skim.
		</p>
		<p>
		</p><pre class="programlisting">
						
OutputStream modelOut = null;
<b class="hl-keyword">try</b> {
  modelOut = <b class="hl-keyword">new</b> BufferedOutputStream(<b class="hl-keyword">new</b> FileOutputStream(modelFile));
  model.serialize(modelOut);
}
<b class="hl-keyword">catch</b> (IOException e) {
  <i class="hl-comment" style="color: silver">// Failed to save model</i>
  e.printStackTrace();
}
<b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (modelOut != null) {
    <b class="hl-keyword">try</b> {
       modelOut.close();
    }
    <b class="hl-keyword">catch</b> (IOException e) {
      <i class="hl-comment" style="color: silver">// Failed to correctly save model.</i>
      <i class="hl-comment" style="color: silver">// Written model might be invalid.</i>
      e.printStackTrace();
    }
  }
}
</pre><p>
		</p>
		</div>
	</div>
</div>
	<div class="chapter" title="Chapter&nbsp;6.&nbsp;Part-of-Speech Tagger"><div class="titlepage"><div><div><h2 class="title"><a name="tools.postagger"></a>Chapter&nbsp;6.&nbsp;Part-of-Speech Tagger</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#tools.postagger.tagging">Tagging</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.postagger.tagging.cmdline">POS Tagger Tool</a></span></dt><dt><span class="section"><a href="#tools.postagger.tagging.api">POS Tagger API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.postagger.training">Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.postagger.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.postagger.training.api">Training API</a></span></dt><dt><span class="section"><a href="#tools.postagger.training.tagdict">Tag Dictionary</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.postagger.eval">Evaluation</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.postagger.eval.tool">Evaluation Tool</a></span></dt></dl></dd></dl></div>

	<div class="section" title="Tagging"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.postagger.tagging"></a>Tagging</h2></div></div></div>
		
		<p>
		The Part of Speech Tagger marks tokens with their corresponding word type
		based on the token itself and the context of the token. A token might have
		multiple pos tags depending on the token and the context. The OpenNLP POS Tagger
		uses a probability model to predict the correct pos tag out of the tag set.
		To limit the possible tags for a token a tag dictionary can be used which increases
		the tagging and runtime performance of the tagger.
		</p>
			<div class="section" title="POS Tagger Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.postagger.tagging.cmdline"></a>POS Tagger Tool</h3></div></div></div>
		
		<p>
		The easiest way to try out the POS Tagger is the command line tool. The tool is
		only intended for demonstration and testing.
		Download the english maxent pos model and start the POS Tagger Tool with this command:
		</p><pre class="screen">
			
$ bin/opennlp POSTagger en-pos-maxent.bin
		 </pre><p>
		The POS Tagger now reads a tokenized sentence per line from stdin.
		Copy these two sentences to the console:
		</p><pre class="programlisting">
			
Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .
Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
		 </pre><p>
		 the POS Tagger will now echo the sentences with pos tags to the console:
		</p><pre class="programlisting">
			
Pierre_NNP Vinken_NNP ,_, 61_CD years_NNS old_JJ ,_, will_MD join_VB the_DT board_NN as_IN
    a_DT nonexecutive_JJ director_NN Nov._NNP 29_CD ._.
Mr._NNP Vinken_NNP is_VBZ chairman_NN of_IN Elsevier_NNP N.V._NNP ,_, the_DT Dutch_NNP publishing_VBG group_NN
		 </pre><p> 
		 The tag set used by the english pos model is the Penn Treebank tag set.
		 See the link below for a description of the tags.
		</p>
      </div>
      
		<div class="section" title="POS Tagger API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.postagger.tagging.api"></a>POS Tagger API</h3></div></div></div>
		
		<p>
		    The POS Tagger can be embedded into an application via its API.
			First the pos model must be loaded into memory from disk or an other source.
			In the sample below its loaded from disk.
			</p><pre class="programlisting">
				
InputStream modelIn = null;

<b class="hl-keyword">try</b> {
  modelIn = <b class="hl-keyword">new</b> FileInputStream(<b class="hl-string"><i style="color:red">"en-pos-maxent.bin"</i></b>);
  POSModel model = <b class="hl-keyword">new</b> POSModel(modelIn);
}
<b class="hl-keyword">catch</b> (IOException e) {
  <i class="hl-comment" style="color: silver">// Model loading failed, handle the error</i>
  e.printStackTrace();
}
<b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (modelIn != null) {
    <b class="hl-keyword">try</b> {
      modelIn.close();
    }
    <b class="hl-keyword">catch</b> (IOException e) {
    }
  }
}
			</pre><p>
			After the model is loaded the POSTaggerME can be instantiated.
			</p><pre class="programlisting">
				
POSTaggerME tagger = <b class="hl-keyword">new</b> POSTaggerME(model);
			</pre><p>
			The POS Tagger instance is now ready to tag data. It expects a tokenized sentence
			as input, which is represented as a String array, each String object in the array
			is one token.
	   </p>
	   <p>
	   The following code shows how to determine the most likely pos tag sequence for a sentence.
	   	</p><pre class="programlisting">
		  
String sent[] = <b class="hl-keyword">new</b> String[]{<b class="hl-string"><i style="color:red">"Most"</i></b>, <b class="hl-string"><i style="color:red">"large"</i></b>, <b class="hl-string"><i style="color:red">"cities"</i></b>, <b class="hl-string"><i style="color:red">"in"</i></b>, <b class="hl-string"><i style="color:red">"the"</i></b>, <b class="hl-string"><i style="color:red">"US"</i></b>, <b class="hl-string"><i style="color:red">"had"</i></b>,
                             <b class="hl-string"><i style="color:red">"morning"</i></b>, <b class="hl-string"><i style="color:red">"and"</i></b>, <b class="hl-string"><i style="color:red">"afternoon"</i></b>, <b class="hl-string"><i style="color:red">"newspapers"</i></b>, <b class="hl-string"><i style="color:red">"."</i></b>};		  
String tags[] = tagger.tag(sent);
			</pre><p>
			The tags array contains one part-of-speech tag for each token in the input array. The corresponding
			tag can be found at the same index as the token has in the input array.
			The confidence scores for the returned tags can be easily retrieved from
			a POSTaggerME with the following method call:
				   	</p><pre class="programlisting">
		  
<b class="hl-keyword">double</b> probs[] = tagger.probs();
			</pre><p>
			The call to probs is stateful and will always return the probabilities of the last
			tagged sentence. The probs method should only be called when the tag method
			was called before, otherwise the behavior is undefined.
			</p>
			<p>
			Some applications need to retrieve the n-best pos tag sequences and not
			only the best sequence.
			The topKSequences method is capable of returning the top sequences.
			It can be called in a similar way as tag.
			</p><pre class="programlisting">
		  
Sequence topSequences[] = tagger.topKSequences(sent);
			</pre><p>	
			Each Sequence object contains one sequence. The sequence can be retrieved
			via Sequence.getOutcomes() which returns a tags array 
			and Sequence.getProbs() returns the probability array for this sequence.
	  		 </p>
	</div>
	</div>
		<div class="section" title="Training"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.postagger.training"></a>Training</h2></div></div></div>
		
		<p>
			The POS Tagger can be trained on annotated training material. The training material
			is a collection of tokenized sentences where each token has the assigned part-of-speech tag.
			The native POS Tagger training material looks like this:
			</p><pre class="programlisting">
		  
About_IN 10_CD Euro_NNP ,_, I_PRP reckon_VBP ._.
That_DT sounds_VBZ good_JJ ._.
			</pre><p>		
			Each sentence must be in one line. The token/tag pairs are combined with "_".
			The token/tag pairs are whitespace separated. The data format does not
			define a document boundary. If a document boundary should be included in the
			training material it is suggested to use an empty line.
		</p>
		<p>The Part-of-Speech Tagger can either be trained with a command line tool,
		or via an trainng API.
		</p>
		
		<div class="section" title="Training Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.postagger.training.tool"></a>Training Tool</h3></div></div></div>
		
		<p>
			OpenNLP has a command line tool which is used to train the models available from the model
			download page on various corpora.
		</p>
		<p>
		 Usage of the tool:
			</p><pre class="screen">
				
$ bin/opennlp POSTaggerTrainer
Usage: opennlp POSTaggerTrainer -lang language -encoding charset [-iterations num] [-cutoff num] \ 
    [-dict tagdict] [-model maxent|perceptron|perceptron_sequence] -data trainingData -model model
-lang language     specifies the language which is being processed.
-encoding charset  specifies the encoding which should be used for reading and writing text.
-iterations num    specified the number of training iterations
-cutoff num        specifies the min number of times a feature must be seen
			 </pre><p>
		</p>
		<p>
		The following command illustrates how an english part-of-speech model can be trained:
		</p><pre class="screen">
		  
$bin/opennlp POSTaggerTrainer -encoding UTF-8 -lang en -model-type maxent -data en-pos.train \ 
-model en-pos-maxent.bin
		 </pre><p>
		</p>
		</div>
		<div class="section" title="Training API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.postagger.training.api"></a>Training API</h3></div></div></div>
		
		<p>
		The Part-of-Speech Tagger training API supports the programmatically training of a new pos model.
		Basically three steps are necessary to train it:
		</p><div class="itemizedlist"><ul class="itemizedlist" type="disc"><li class="listitem">
				<p>The application must open a sample data stream</p>
			</li><li class="listitem">
				<p>Call the POSTagger.train method</p>
			</li><li class="listitem">
				<p>Save the POSModel to a file or database</p>
			</li></ul></div><p>
		The following code illustrates that:
		</p><pre class="programlisting">
				
POSModel model = null;

InputStream dataIn = null;
<b class="hl-keyword">try</b> {
  dataIn = <b class="hl-keyword">new</b> FileInputStream(<b class="hl-string"><i style="color:red">"en-pos.train"</i></b>);
  ObjectStream&lt;String&gt; lineStream =
		<b class="hl-keyword">new</b> PlainTextByLineStream(dataIn, <b class="hl-string"><i style="color:red">"UTF-8"</i></b>);
  ObjectStream&lt;POSSample&gt; sampleStream = <b class="hl-keyword">new</b> WordTagSampleStream(lineStream);

  model = POSTaggerME.train(<b class="hl-string"><i style="color:red">"en"</i></b>, sampleStream, ModelType.MAXENT,
      null, null, <span class="hl-number">100</span>, <span class="hl-number">5</span>);
}
<b class="hl-keyword">catch</b> (IOException e) {
  <i class="hl-comment" style="color: silver">// Failed to read or parse training data, training failed</i>
  e.printStackTrace();
}
<b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (dataIn != null) {
    <b class="hl-keyword">try</b> {
      dataIn.close();
    }
    <b class="hl-keyword">catch</b> (IOException e) {
      <i class="hl-comment" style="color: silver">// Not an issue, training already finished.</i>
      <i class="hl-comment" style="color: silver">// The exception should be logged and investigated</i>
      <i class="hl-comment" style="color: silver">// if part of a production system.</i>
      e.printStackTrace();
    }
  }
}
	</pre><p>
	The above code performs the first two steps, opening the data and training
	the model. The trained model must still be saved into an OutputStream, in
	the sample below it is written into a file.
	</p><pre class="programlisting">
				
OutputStream modelOut = null;
<b class="hl-keyword">try</b> {
  modelOut = <b class="hl-keyword">new</b> BufferedOutputStream(<b class="hl-keyword">new</b> FileOutputStream(modelFile));
  model.serialize(modelOut);
}
<b class="hl-keyword">catch</b> (IOException e) {
  <i class="hl-comment" style="color: silver">// Failed to save model</i>
  e.printStackTrace();
}
<b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (modelOut != null) {
  <b class="hl-keyword">try</b> {
     modelOut.close();
  }
  <b class="hl-keyword">catch</b> (IOException e) {
    <i class="hl-comment" style="color: silver">// Failed to correctly save model.</i>
    <i class="hl-comment" style="color: silver">// Written model might be invalid.</i>
    e.printStackTrace();
  }
}
		</pre><p>
		</p>
		</div>
		<div class="section" title="Tag Dictionary"><div class="titlepage"><div><div><h3 class="title"><a name="tools.postagger.training.tagdict"></a>Tag Dictionary</h3></div></div></div>
		
		<p>
		The tag dicitionary is a word dictionary which specifies which tags a specific token can have. Using a tag
		dictionary has two advantages, unappropriate tags can not been assigned to tokens in the dictionary and the
		beam search algrotihm has to consider less possibilties and can search faster.
		</p>
		<p>
		The dictionary is defined in a xml format and can be created and stored with the POSDictionary class.
		Pleaes for now checkout the javadoc and source code of that class.
		</p>
		<p>Note: The format should be documented and sample code should show how to use the dictionary.
			  Any contributions are very welcome. If you want to contribute please contact us on the mailing list
			  or comment on the jira issue <a class="ulink" href="https://issues.apache.org/jira/browse/OPENNLP-287" target="_top">OPENNLP-287</a>.
		</p>
		</div>
		</div>
		
		<div class="section" title="Evaluation"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.postagger.eval"></a>Evaluation</h2></div></div></div>
		
		<p>
		The built in evaluation can measure the accuracy of the pos tagger.
		The accuracy can be measured on a test data set or via cross validation.
		</p>
		<div class="section" title="Evaluation Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.postagger.eval.tool"></a>Evaluation Tool</h3></div></div></div>
		
		<p>
		There is a command line tool to evaluate a given model on a test data set.
		The command line tool currently does not support the cross validation
		evaluation (contribution welcome).
		The following command shows how the tool can be run:
		</p><pre class="screen">
				
$bin/opennlp POSTaggerEvaluator -encoding utf-8 -model pt.postagger.model -data pt.postagger.test
			 </pre><p>
			 This will display the resulting accuracy score, e.g.:
			 </p><pre class="screen">
				
Loading model ... done
Evaluating ... done

Accuracy: 0.9659110277825124
			 </pre><p>
		</p> 
		</div>
		</div>
</div>
	<div class="chapter" title="Chapter&nbsp;7.&nbsp;Chunker"><div class="titlepage"><div><div><h2 class="title"><a name="tools.chunker"></a>Chapter&nbsp;7.&nbsp;Chunker</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#tools.parser.chunking">Chunking</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.parser.chunking.cmdline">Chunker Tool</a></span></dt><dt><span class="section"><a href="#tools.parser.chunking.api">Chunking API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.chunker.training">Chunker Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.chunker.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.chunker.training.api">Training API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.chunker.evaluation">Chunker Evaluation</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.chunker.evaluation.tool">Chunker Evaluation Tool</a></span></dt></dl></dd></dl></div>

	

	<div class="section" title="Chunking"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.parser.chunking"></a>Chunking</h2></div></div></div>
		
		<p>
		Text chunking consists of dividing a text in syntactically correlated parts of words,
		like noun groups, verb groups, but does not specify their internal structure, nor their role in the main sentence. 
		</p>
		
		<div class="section" title="Chunker Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.parser.chunking.cmdline"></a>Chunker Tool</h3></div></div></div>
		
		<p>
		The easiest way to try out the Chunker is the command line tool. The tool is only intended
		for demonstration and testing.
		</p> 
		<p>
		Download the english maxent chunker model from the website and start the Chunker Tool with this command:
		</p>
		<p>
				</p><pre class="programlisting">
				
bin/opennlp ChunkerME en-chunker.bin
		</pre><p>
		The Chunker now reads a pos tagged sentence per line from stdin.
		Copy these two sentences to the console: 
		</p><pre class="programlisting">
				
Rockwell_NNP International_NNP Corp._NNP 's_POS Tulsa_NNP unit_NN said_VBD it_PRP signed_VBD 
    a_DT tentative_JJ agreement_NN extending_VBG its_PRP$ contract_NN with_IN Boeing_NNP Co._NNP
    to_TO provide_VB structural_JJ parts_NNS for_IN Boeing_NNP 's_POS 747_CD jetliners_NNS ._.
Rockwell_NNP said_VBD the_DT agreement_NN calls_VBZ for_IN it_PRP to_TO supply_VB 200_CD
    additional_JJ so-called_JJ shipsets_NNS for_IN the_DT planes_NNS ._.
		</pre><p>
		the Chunker will now echo the sentences grouped tokens to the console:
				</p><pre class="programlisting">
				
[NP Rockwell_NNP International_NNP Corp._NNP ] [NP 's_POS Tulsa_NNP unit_NN ] [VP said_VBD ]
    [NP it_PRP ] [VP signed_VBD ] [NP a_DT tentative_JJ agreement_NN ] [VP extending_VBG ]
    [NP its_PRP$ contract_NN ] [PP with_IN ] [NP Boeing_NNP Co._NNP ] [VP to_TO provide_VB ]
    [NP structural_JJ parts_NNS ] [PP for_IN ] [NP Boeing_NNP ] [NP 's_POS 747_CD jetliners_NNS ] ._.
[NP Rockwell_NNP ] [VP said_VBD ] [NP the_DT agreement_NN ] [VP calls_VBZ ] [SBAR for_IN ]
    [NP it_PRP ] [VP to_TO supply_VB ] [NP 200_CD additional_JJ so-called_JJ shipsets_NNS ]
    [PP for_IN ] [NP the_DT planes_NNS ] ._.
		</pre><p>
		The tag set used by the english pos model is the Penn Treebank tag set. 
		See the link below for a description of the tags.
		</p>
		</div>
		<div class="section" title="Chunking API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.parser.chunking.api"></a>Chunking API</h3></div></div></div>
		
		<p>
		    The Chunker can be embedded into an application via its API.
			First the chunker model must be loaded into memory from disk or an other source.
			In the sample below its loaded from disk.
			</p><pre class="programlisting">
				
InputStream modelIn = null;
ChunkerModel model = null;

<b class="hl-keyword">try</b> {
  modelIn = <b class="hl-keyword">new</b> FileInputStream(<b class="hl-string"><i style="color:red">"en-chunker.bin"</i></b>);
  model = <b class="hl-keyword">new</b> ChunkerModel(modelIn);
} <b class="hl-keyword">catch</b> (IOException e) {
  <i class="hl-comment" style="color: silver">// Model loading failed, handle the error</i>
  e.printStackTrace();
} <b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (modelIn != null) {
    <b class="hl-keyword">try</b> {
      modelIn.close();
    } <b class="hl-keyword">catch</b> (IOException e) {
    }
  }
}
			</pre><p>
			After the model is loaded a Chunker can be instantiated.
			</p><pre class="programlisting">
				
ChunkerME chunker = <b class="hl-keyword">new</b> ChunkerME(model);
			</pre><p>
			The Chunker instance is now ready to tag data. It expects a tokenized sentence
			as input, which is represented as a String array, each String object in the array
			is one token, and the POS tags associated with each token.
	   </p>
	   <p>
	   The following code shows how to determine the most likely chunk tag sequence for a sentence.
	   	</p><pre class="programlisting">
		  
String sent[] = <b class="hl-keyword">new</b> String[] { <b class="hl-string"><i style="color:red">"Rockwell"</i></b>, <b class="hl-string"><i style="color:red">"International"</i></b>, <b class="hl-string"><i style="color:red">"Corp."</i></b>, <b class="hl-string"><i style="color:red">"'s"</i></b>,
    <b class="hl-string"><i style="color:red">"Tulsa"</i></b>, <b class="hl-string"><i style="color:red">"unit"</i></b>, <b class="hl-string"><i style="color:red">"said"</i></b>, <b class="hl-string"><i style="color:red">"it"</i></b>, <b class="hl-string"><i style="color:red">"signed"</i></b>, <b class="hl-string"><i style="color:red">"a"</i></b>, <b class="hl-string"><i style="color:red">"tentative"</i></b>, <b class="hl-string"><i style="color:red">"agreement"</i></b>,
    <b class="hl-string"><i style="color:red">"extending"</i></b>, <b class="hl-string"><i style="color:red">"its"</i></b>, <b class="hl-string"><i style="color:red">"contract"</i></b>, <b class="hl-string"><i style="color:red">"with"</i></b>, <b class="hl-string"><i style="color:red">"Boeing"</i></b>, <b class="hl-string"><i style="color:red">"Co."</i></b>, <b class="hl-string"><i style="color:red">"to"</i></b>,
    <b class="hl-string"><i style="color:red">"provide"</i></b>, <b class="hl-string"><i style="color:red">"structural"</i></b>, <b class="hl-string"><i style="color:red">"parts"</i></b>, <b class="hl-string"><i style="color:red">"for"</i></b>, <b class="hl-string"><i style="color:red">"Boeing"</i></b>, <b class="hl-string"><i style="color:red">"'s"</i></b>, <b class="hl-string"><i style="color:red">"747"</i></b>,
    <b class="hl-string"><i style="color:red">"jetliners"</i></b>, <b class="hl-string"><i style="color:red">"."</i></b> };

String pos[] = <b class="hl-keyword">new</b> String[] { <b class="hl-string"><i style="color:red">"NNP"</i></b>, <b class="hl-string"><i style="color:red">"NNP"</i></b>, <b class="hl-string"><i style="color:red">"NNP"</i></b>, <b class="hl-string"><i style="color:red">"POS"</i></b>, <b class="hl-string"><i style="color:red">"NNP"</i></b>, <b class="hl-string"><i style="color:red">"NN"</i></b>,
    <b class="hl-string"><i style="color:red">"VBD"</i></b>, <b class="hl-string"><i style="color:red">"PRP"</i></b>, <b class="hl-string"><i style="color:red">"VBD"</i></b>, <b class="hl-string"><i style="color:red">"DT"</i></b>, <b class="hl-string"><i style="color:red">"JJ"</i></b>, <b class="hl-string"><i style="color:red">"NN"</i></b>, <b class="hl-string"><i style="color:red">"VBG"</i></b>, <b class="hl-string"><i style="color:red">"PRP$"</i></b>, <b class="hl-string"><i style="color:red">"NN"</i></b>, <b class="hl-string"><i style="color:red">"IN"</i></b>,
    <b class="hl-string"><i style="color:red">"NNP"</i></b>, <b class="hl-string"><i style="color:red">"NNP"</i></b>, <b class="hl-string"><i style="color:red">"TO"</i></b>, <b class="hl-string"><i style="color:red">"VB"</i></b>, <b class="hl-string"><i style="color:red">"JJ"</i></b>, <b class="hl-string"><i style="color:red">"NNS"</i></b>, <b class="hl-string"><i style="color:red">"IN"</i></b>, <b class="hl-string"><i style="color:red">"NNP"</i></b>, <b class="hl-string"><i style="color:red">"POS"</i></b>, <b class="hl-string"><i style="color:red">"CD"</i></b>, <b class="hl-string"><i style="color:red">"NNS"</i></b>,
    <b class="hl-string"><i style="color:red">"."</i></b> };

String tag[] = chunker.chunk(sent, pos);
			</pre><p>
			The tags array contains one chunk tag for each token in the input array. The corresponding
			tag can be found at the same index as the token has in the input array.
			The confidence scores for the returned tags can be easily retrieved from
			a ChunkerME with the following method call:
				   	</p><pre class="programlisting">
		  
<b class="hl-keyword">double</b> probs[] = chunker.probs();
			</pre><p>
			The call to probs is stateful and will always return the probabilities of the last
			tagged sentence. The probs method should only be called when the tag method
			was called before, otherwise the behavior is undefined.
			</p>
			<p>
			Some applications need to retrieve the n-best chunk tag sequences and not
			only the best sequence.
			The topKSequences method is capable of returning the top sequences.
			It can be called in a similar way as chunk.
			</p><pre class="programlisting">
		  
Sequence topSequences[] = chunk.topKSequences(sent, pos);
			</pre><p>	
			Each Sequence object contains one sequence. The sequence can be retrieved
			via Sequence.getOutcomes() which returns a tags array 
			and Sequence.getProbs() returns the probability array for this sequence.
	  		 </p>
		</div>
	</div>
	<div class="section" title="Chunker Training"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.chunker.training"></a>Chunker Training</h2></div></div></div>
		
		<p>
		The pre-trained models might not be available for a desired language,
		can not detect important entities or the performance is not good enough outside the news domain.
		</p>
		<p>
		These are the typical reason to do custom training of the chunker on a ne
	    corpus or on a corpus which is extended by private training data taken from the data which should be analyzed.
		</p>
		<p>
		The training data must be converted to the OpenNLP chunker training format,
		that is based on <a class="ulink" href="http://www.cnts.ua.ac.be/conll2000/chunking" target="_top">CoNLL2000</a>:
		The train data consist of three columns separated by spaces. Each word has been put on a
		separate line and there is an empty line after each sentence. The first column contains
		the current word, the second its part-of-speech tag and the third its chunk tag. 
		The chunk tags contain the name of the chunk type, for example I-NP for noun phrase words
		and I-VP for verb phrase words. Most chunk types have two types of chunk tags,
		B-CHUNK for the first word of the chunk and I-CHUNK for each other word in the chunk.
		Here is an example of the file format:
		</p>
		<p>
		Sample sentence of the training data: 
		</p><pre class="programlisting">
				
He        PRP  B-NP
reckons   VBZ  B-VP
the       DT   B-NP
current   JJ   I-NP
account   NN   I-NP
deficit   NN   I-NP
will      MD   B-VP
narrow    VB   I-VP
to        TO   B-PP
only      RB   B-NP
#         #    I-NP
1.8       CD   I-NP
billion   CD   I-NP
in        IN   B-PP
September NNP  B-NP
.         .    O
		</pre><p>
		</p>
		<div class="section" title="Training Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.chunker.training.tool"></a>Training Tool</h3></div></div></div>
		
		<p>
		OpenNLP has a command line tool which is used to train the models available from the
		model download page on various corpora.
		</p> 
		<p>
		Usage of the tool:
				</p><pre class="programlisting">
				
$ bin/opennlp ChunkerTrainerME
Usage: opennlp ChunkerTrainerME-lang language -encoding charset [-iterations num] \
[-cutoff num] -data trainingData -model model
-lang language     specifies the language which is being processed.
-encoding charset  specifies the encoding which should be used for reading and writing text.
-iterations num    specified the number of training iterations
-cutoff num        specifies the min number of times a feature must be seen
		</pre><p>
		Its now assumed that the english chunker model should be trained from a file called
		en-chunker.train which is encoded as UTF-8. The following command will train the
		name finder and write the model to en-chunker.bin: 
		</p><pre class="programlisting">
		
bin/opennlp ChunkerTrainerME -encoding UTF-8 -lang en -data en-chunker.train -model en-chunker.bin
		</pre><p>
		Additionally its possible to specify the number of iterations, the cutoff and to overwrite
		all types in the training data with a single type.
		</p>
		</div>
				<div class="section" title="Training API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.chunker.training.api"></a>Training API</h3></div></div></div>
		
		<p>TODO: Write documentation about the chunker training api. Any contributions
			  are very welcome. If you want to contribute please contact us on the mailing list
			  or comment on the jira issue <a class="ulink" href="https://issues.apache.org/jira/browse/OPENNLP-218" target="_top">OPENNLP-218</a>.
		</p>
		</div>
	</div>
	
	<div class="section" title="Chunker Evaluation"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.chunker.evaluation"></a>Chunker Evaluation</h2></div></div></div>
		
		<p>
		The built in evaluation can measure the chunker performance. The performance is either
		measured on a test dataset or via cross validation. 
		</p>
		<div class="section" title="Chunker Evaluation Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.chunker.evaluation.tool"></a>Chunker Evaluation Tool</h3></div></div></div>
		
		<p>
		The following command shows how the tool can be run:
				</p><pre class="programlisting">
				
bin/opennlp ChunkerEvaluator
Usage: opennlp ChunkerEvaluator [-encoding charsetName] -data data -model model
		</pre><p>
		A sample of the command considering you have a data sample named en-chunker.eval
		and you trainned a model called en-chunker.bin:
				</p><pre class="programlisting">
				
bin/opennlp ChunkerEvaluator -lang en -encoding UTF-8 -data en-chunker.eval -model en-chunker.bin
		</pre><p>		
		and here is a sample output:  
		</p><pre class="programlisting">
		
Precision: 0.9255923572240226
Recall: 0.9220610430991112
F-Measure: 0.9238233255623465
		</pre><p>
		You can also use the tool to perform 10-fold cross validation of the Chunker.
he following command shows how the tool can be run:
				</p><pre class="programlisting">
				
bin/opennlp ChunkerCrossValidator
Usage: opennlp ChunkerCrossValidator -lang language -encoding charset [-iterations num] [-cutoff num]
-lang language     specifies the language which is being processed.
-encoding charset  specifies the encoding which should be used for reading and writing text.
-iterations num    specified the number of training iterations
-cutoff num        specifies the min number of times a feature must be seen
-data trainingData      training data used for cross validation
		</pre><p>
		It is not necessary to pass a model. The tool will automatically split the data to train and evaluate:
				</p><pre class="programlisting">
				
bin/opennlp ChunkerCrossValidator -lang pt -encoding UTF-8 -data en-chunker.cross
		</pre><p>
		</p>
		</div>
	</div>
</div>
	<div class="chapter" title="Chapter&nbsp;8.&nbsp;Parser"><div class="titlepage"><div><div><h2 class="title"><a name="tools.parser"></a>Chapter&nbsp;8.&nbsp;Parser</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#tools.parser.parsing">Parsing</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.parser.parsing.cmdline">Parser Tool</a></span></dt><dt><span class="section"><a href="#tools.parser.parsing.api">Parsing API</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.parser.training">Parser Training</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.parser.training.tool">Training Tool</a></span></dt><dt><span class="section"><a href="#tools.parser.training.api">Training API</a></span></dt></dl></dd></dl></div>

	

	<div class="section" title="Parsing"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.parser.parsing"></a>Parsing</h2></div></div></div>
		
		<p>
		</p>
		
		<div class="section" title="Parser Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.parser.parsing.cmdline"></a>Parser Tool</h3></div></div></div>
		
		<p>
		The easiest way to try out the Parser is the command line tool.
		The tool is only intended for demonstration and testing.
		Download the english chunking parser model from the our website and start the Parse
 		Tool with the following command.
				</p><pre class="programlisting">
				
$bin/opennlp Parser en-parser.bin en-parser-chunking.bin
		</pre><p>
		Loading the big parser model can take several seconds, be patient.
		Copy this sample sentence to the console.
		</p><pre class="programlisting">
				
The quick brown fox jumps over the lazy dog .
		</pre><p>
		The parser should now print the following to the console.
				</p><pre class="programlisting">
				
(TOP (NP (NP (DT The) (JJ quick) (JJ brown) (NN fox) (NNS jumps)) (PP (IN over) (NP (DT the)
    (JJ lazy) (NN dog))) (. .)))
		</pre><p>
		With the following command the input can be read from a file and be written to an output file.
				</p><pre class="programlisting">
				
$ bin/opennlp Parser en-parser.bin en-parser-chunking.bin &lt; article-tokenized.txt &gt; article-parsed.txt.
		</pre><p>
		The article-tokenized.txt file must contain one sentence per line which is
		tokenized with the english tokenizer model from our website.
		See the Tokenizer documentation for further details.
		</p>
		</div>
		<div class="section" title="Parsing API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.parser.parsing.api"></a>Parsing API</h3></div></div></div>
		
		<p>
			The Parser can be easily integrated into an application via its API.
			To instantiate a Parser the parser model must be loaded first.
			</p><pre class="programlisting">
				
InputStream modelIn = <b class="hl-keyword">new</b> FileInputStream(<b class="hl-string"><i style="color:red">"en-parser-chunking.bin"</i></b>);
<b class="hl-keyword">try</b> {
  ParserModel model = <b class="hl-keyword">new</b> ParserModel(modelIn);
}
<b class="hl-keyword">catch</b> (IOException e) {
  e.printStackTrace();
}
<b class="hl-keyword">finally</b> {
  <b class="hl-keyword">if</b> (modelIn != null) {
    <b class="hl-keyword">try</b> {
      modelIn.close();
    }
    <b class="hl-keyword">catch</b> (IOException e) {
    }
  }
}
		</pre><p>
		Unlike the other components to instantiate the Parser a factory method
		should be used instead of creating the Parser via the new operator.
		The parser model is either trained for the chunking parser or the tree
		insert parser the parser implementation must be chosen correctly.
		The factory method will read a type parameter from the model and create
		an instance of the corresponding parser implementation.
		</p><pre class="programlisting">
				
Parser parser = ParserFactory.create(model);
		</pre><p>
		Right now the tree insert parser is still experimental and there is no pre-trained model for it.
		The parser expect a whitespace tokenized sentence. A utility method from the command
		line tool can parse the sentence String. The following code shows how the parser can be called.
				</p><pre class="programlisting">
				
String sentence = <b class="hl-string"><i style="color:red">"The quick brown fox jumps over the lazy dog ."</i></b>;
Parse topParses[] = ParserTool.parseLine(sentence, parser, <span class="hl-number">1</span>);
		</pre><p>
		
		The topParses array only contains one parse because the number of parses is set to 1.
		The Parse object contains the parse tree.
		To display the parse tree call the show method. It either prints the parse to
		the console or into a provided StringBuffer. Similar to Exception.printStackTrace.
		</p>
		<p>
		TODO: Extend this section with more information about the Parse object.
		</p>
		</div>
	</div>
	<div class="section" title="Parser Training"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.parser.training"></a>Parser Training</h2></div></div></div>
		
		<p>
				The OpenNLP offers two different parser implementations, the chunking parser and the
		treeinsert parser. The later one is still experimental and not recommended for production use.
		(TODO: Add a section which explains the two different approches)
		The training can either be done with the command line tool or the training API.
		In the first case the training data must be available in the OpenNLP format. Which is
		the Penn Treebank format, but with the limitation of a sentence per line.
		</p><pre class="programlisting">
				
(TOP (S (NP-SBJ (DT Some) )(VP (VBP say) (NP (NNP November) ))(. .) ))
(TOP (S (NP-SBJ (PRP I) )(VP (VBP say) (NP (CD 1992) ))(. .) ('' '') ))
		</pre><p>
		(TODO: Insert link which explains the penn treebank format.)
		A parser model also contains a pos tagger model, depending on the amount of available
		training data it is recommend to switch the tagger model against a tagger model which
		was trained on a larger corpus. The pre-trained parser model provided on the website
		is doing this to achieve a better performance. (TODO: On which data is the model on
		the website trained, and say on which data the tagger model is trained)
		</p>
		<div class="section" title="Training Tool"><div class="titlepage"><div><div><h3 class="title"><a name="tools.parser.training.tool"></a>Training Tool</h3></div></div></div>
		
		<p>
		OpenNLP has a command line tool which is used to train the models available from the
		model download page on various corpora. The data must be converted to the OpenNLP parser
		training format, which is shortly explained above.
		To train the parser a head rules file is also needed. (TODO: Add documentation about the head rules file)
		Usage of the tool:
				</p><pre class="programlisting">
				
$ bin/opennlp ParserTrainer
Usage: opennlp ParserTrainer-lang language -encoding charset [-iterations num] \ 
[-cutoff num] -head-rules head_rules -data trainingData -model model
-lang language     specifies the language which is being processed.
-encoding charset  specifies the encoding which should be used for reading and writing text.
-iterations num    specified the number of training iterations
-cutoff num        specifies the min number of times a feature must be seen
		</pre><p>
		The model on the website was trained with the following command:
		</p><pre class="programlisting">
		
$bin/opennlp ParserTrainer -encoding ISO-8859-1 -lang en -parserType CHUNKING -head-rules head_rules \
    -data train.all -model en-parser-chunking.bin
		</pre><p>
		Its also possible to specify the cutoff and the number of iterations, these parameters
		are used for all trained models. The -parserType parameter is an optional parameter,
		to use the tree insertion parser, specify TREEINSERT as type. The TaggerModelReplacer
		tool replaces the tagger model inside the parser model with a new one. 
		</p>
		<p>
		Note: The original parser model will be overwritten with the new parser model which
		contains the replaced tagger model.
				</p><pre class="programlisting">
		
$bin/opennlp TaggerModelReplacer  models/en-parser-chunking.bin models/en-pos-maxent.bin
		</pre><p>
		Additionally there are tools to just retrain the build or the check model.
		</p>
		</div>
		<div class="section" title="Training API"><div class="titlepage"><div><div><h3 class="title"><a name="tools.parser.training.api"></a>Training API</h3></div></div></div>
		
		<p>TODO: Write documentation about the parser training api. Any contributions
			  are very welcome. If you want to contribute please contact us on the mailing list
			  or comment on the jira issue <a class="ulink" href="https://issues.apache.org/jira/browse/OPENNLP-219" target="_top">OPENNLP-219</a>.
		</p>
		</div>
	</div>
</div>
	<div class="chapter" title="Chapter&nbsp;9.&nbsp;Coreference Resolution"><div class="titlepage"><div><div><h2 class="title"><a name="tools.coref"></a>Chapter&nbsp;9.&nbsp;Coreference Resolution</h2></div></div></div>

<p>TODO: Write documentation about the coref component. Any contributions
are very welcome. If you want to contribute please contact us on the mailing list
or comment on the jira issue <a class="ulink" href="https://issues.apache.org/jira/browse/OPENNLP-48" target="_top">OPENNLP-48</a>.</p>
</div>
	<div class="chapter" title="Chapter&nbsp;10.&nbsp;Corpora"><div class="titlepage"><div><div><h2 class="title"><a name="tools.corpora"></a>Chapter&nbsp;10.&nbsp;Corpora</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#tools.corpora.conll">CONLL</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.corpora.conll.2000">CONLL 2000</a></span></dt><dt><span class="section"><a href="#tools.corpora.conll.2003">CONLL 2002</a></span></dt><dt><span class="section"><a href="#tools.corpora.conll.2003">CONLL 2003</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.corpora.arvores-deitadas">Arvores Deitadas</a></span></dt><dd><dl><dt><span class="section"><a href="#tools.corpora.arvores-deitadas.getting">Getting the data</a></span></dt><dt><span class="section"><a href="#tools.corpora.arvores-deitadas.converting">Converting the data</a></span></dt><dt><span class="section"><a href="#tools.corpora.arvores-deitadas.evaluation">Evaluation</a></span></dt></dl></dd><dt><span class="section"><a href="#tools.corpora.leipzig">Leipzig Corpora</a></span></dt></dl></div>

	
	<p>
	OpenNLP has built-in support to convert various corpora
	into the native training format needed by the different
	trainable components.
	</p>
	<div class="section" title="CONLL"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.corpora.conll"></a>CONLL</h2></div></div></div>
		
		<p>
		CoNLL stands for the Confernece on Computational Natural Language Learning and is not
		a single project but a consortium of developers attempting to broaden the computing
		environment. More information about the entire conference series can be obtained here
		for CoNLL.
		</p>
		<div class="section" title="CONLL 2000"><div class="titlepage"><div><div><h3 class="title"><a name="tools.corpora.conll.2000"></a>CONLL 2000</h3></div></div></div>
		
		<p>
		The shared task of CoNLL-2000 is Chunking .
		</p>
		<div class="section" title="Getting the data"><div class="titlepage"><div><div><h4 class="title"><a name="tools.corpora.conll.2000.getting"></a>Getting the data</h4></div></div></div>
		
		<p>
		CoNLL-2000 made available training and test data for the Chunk task in English. 
		The data consists of the same partitions of the Wall Street Journal corpus (WSJ) 
		as the widely used data for noun phrase chunking: sections 15-18 as training data 
		(211727 tokens) and section 20 as test data (47377 tokens). The annotation of the 
		data has been derived from the WSJ corpus by a program written by Sabine Buchholz 
		from Tilburg University, The Netherlands. Both training and test data can be
		obtained from <a class="ulink" href="http://www.cnts.ua.ac.be/conll2000/chunking" target="_top">http://www.cnts.ua.ac.be/conll2000/chunking</a>. 
		</p>
		</div>
		<div class="section" title="Converting the data"><div class="titlepage"><div><div><h4 class="title"><a name="tools.corpora.conll.2000.converting"></a>Converting the data</h4></div></div></div>
		
		<p>
		The data don't need to be transformed because Apache OpenNLP Chunker follows
		the CONLL 2000 format for training. Check <a class="link" href="#tools.chunker.training" title="Chunker Training">Chunker Training</a> section to learn more.
		</p>
		</div>
		<div class="section" title="Training"><div class="titlepage"><div><div><h4 class="title"><a name="tools.corpora.conll.2000.training"></a>Training</h4></div></div></div>
		
		<p>
		 We can train the model for the Chunker using the train.txt available at CONLL 2000:
		 </p><pre class="programlisting">
			
bin/opennlp ChunkerTrainerME -encoding UTF-8 -lang en -iterations 500 \
-data train.txt -model en-chunker.bin
		</pre><p>
		</p><pre class="programlisting">
			
Indexing events using cutoff of 5

	Computing event counts...  done. 211727 events
	Indexing...  done.
Sorting and merging events... done. Reduced 211727 events to 197252.
Done indexing.
Incorporating indexed data for training...  
done.
	Number of Event Tokens: 197252
	    Number of Outcomes: 22
	  Number of Predicates: 107838
...done.
Computing model parameters...
Performing 500 iterations.
  1:  .. loglikelihood=-654457.1455212828	0.2601510435608118
  2:  .. loglikelihood=-239513.5583724216	0.9260037690044255
  3:  .. loglikelihood=-141313.1386347238	0.9443387003074715
  4:  .. loglikelihood=-101083.50853437989	0.954375209585929
... cut lots of iterations ...
498:  .. loglikelihood=-1710.8874647317095	0.9995040783650645
499:  .. loglikelihood=-1708.0908900815848	0.9995040783650645
500:  .. loglikelihood=-1705.3045902366732	0.9995040783650645
Writing chunker model ... done (4.019s)

Wrote chunker model to path: .\en-chunker.bin
		</pre><p>
		</p>
		</div>
		<div class="section" title="Evaluating"><div class="titlepage"><div><div><h4 class="title"><a name="tools.corpora.conll.2000.evaluation"></a>Evaluating</h4></div></div></div>
		
		<p>
		We evaluate the model using the file test.txt  available at CONLL 2000:
		</p><pre class="programlisting">
			
$ bin/opennlp ChunkerEvaluator -encoding utf8 -model en-chunker.bin -data test.txt
		</pre><p>
		</p><pre class="programlisting">
			
Loading Chunker model ... done (0,665s)
current: 85,8 sent/s avg: 85,8 sent/s total: 86 sent
current: 88,1 sent/s avg: 87,0 sent/s total: 174 sent
current: 156,2 sent/s avg: 110,0 sent/s total: 330 sent
current: 192,2 sent/s avg: 130,5 sent/s total: 522 sent
current: 167,2 sent/s avg: 137,8 sent/s total: 689 sent
current: 179,2 sent/s avg: 144,6 sent/s total: 868 sent
current: 183,2 sent/s avg: 150,3 sent/s total: 1052 sent
current: 183,2 sent/s avg: 154,4 sent/s total: 1235 sent
current: 169,2 sent/s avg: 156,0 sent/s total: 1404 sent
current: 178,2 sent/s avg: 158,2 sent/s total: 1582 sent
current: 172,2 sent/s avg: 159,4 sent/s total: 1754 sent
current: 177,2 sent/s avg: 160,9 sent/s total: 1931 sent


Average: 161,6 sent/s 
Total: 2013 sent
Runtime: 12.457s

Precision: 0.9244354736974896
Recall: 0.9216837162502096
F-Measure: 0.9230575441395671
		</pre><p>
		</p>
		</div>
	</div>
		<div class="section" title="CONLL 2002"><div class="titlepage"><div><div><h3 class="title"><a name="tools.corpora.conll.2003"></a>CONLL 2002</h3></div></div></div>
		
		<p>
		TODO: Document how to use the converters for CONLL 2002. Any contributions
		are very welcome. If you want to contribute please contact us on the mailing list
		or comment on the jira issue 
		<a class="ulink" href="https://issues.apache.org/jira/browse/OPENNLP-46" target="_top">OPENNLP-46</a>.
		</p>
		</div>
		<div class="section" title="CONLL 2003"><div class="titlepage"><div><div><h3 class="title"><a name="tools.corpora.conll.2003"></a>CONLL 2003</h3></div></div></div>
		
		<p>
		The shared task of CoNLL-2003 is language independent named entity recognition
		for English and German.
		</p>
		<div class="section" title="Getting the data"><div class="titlepage"><div><div><h4 class="title"><a name="tools.corpora.conll.2003.getting"></a>Getting the data</h4></div></div></div>
		
		<p>
		The English data is the Reuters Corpus, which is a collection of news wire articles.
		The Reuters Corpus can be obtained free of charges from the NIST for research
		purposes: <a class="ulink" href="http://trec.nist.gov/data/reuters/reuters.html" target="_top">http://trec.nist.gov/data/reuters/reuters.html</a>
		</p>
		<p>
		The German data is a collection of articles from the German newspaper Frankfurter
		Rundschau. The articles are part of the ECI Multilingual Text Corpus which
		can be obtained for 75$ (2010) from the Linguistic Data Consortium:
<a class="ulink" href="http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC94T5" target="_top">http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC94T5</a>		</p>
		<p>After one of the corpora is available the data must be
		transformed as explained in the README file to the conll format.
		The transformed data can be read by the OpenNLP CONLL03 converter.
		</p>
		</div>
		<div class="section" title="Converting the data"><div class="titlepage"><div><div><h4 class="title"><a name="tools.corpora.conll.2003.converting"></a>Converting the data</h4></div></div></div>
		
		<p>
		To convert the information to the OpenNLP format:
		</p><pre class="programlisting">
			
$ bin/opennlp TokenNameFinderConverter conll03 -data eng.train -lang en -types per &gt; corpus_train.txt
		</pre><p>
		Optionally, you can convert the training test samples as well.
		</p><pre class="programlisting">
			
bin/opennlp TokenNameFinderConverter conll03 -data eng.testa -lang en -types per &gt; corpus_testa.txt
bin/opennlp TokenNameFinderConverter conll03 -data eng.testb -lang en -types per &gt; corpus_testb.txt
		</pre><p>
		</p>
		</div>
		<div class="section" title="Training with English data"><div class="titlepage"><div><div><h4 class="title"><a name="tools.corpora.conll.2003.training.english"></a>Training with English data</h4></div></div></div>
		
		<p>
		 To train the model for the name finder:
		 </p><pre class="programlisting">
			
$ bin/opennlp TokenNameFinderTrainer -lang en -encoding utf8 -iterations 500 \
    -data corpus_train.txt -model en_ner_person.bin
		</pre><p>
		</p><pre class="programlisting">
			
Indexing events using cutoff of 5

	Computing event counts...  done. 203621 events
	Indexing...  done.
Sorting and merging events... done. Reduced 203621 events to 179409.
Done indexing.
Incorporating indexed data for training...  
done.
	Number of Event Tokens: 179409
	    Number of Outcomes: 3
	  Number of Predicates: 58814
...done.
Computing model parameters...
Performing 500 iterations.
  1:  .. loglikelihood=-223700.5328318588	0.9453494482396216
  2:  .. loglikelihood=-40525.939777363084	0.9467933071736215
  3:  .. loglikelihood=-24893.98837874921	0.9598518816821447
  4:  .. loglikelihood=-18420.3379471033	0.9712996203731442
... cut lots of iterations ...
498:  .. loglikelihood=-952.8501399442295	0.9988950059178572
499:  .. loglikelihood=-952.0600155746948	0.9988950059178572
500:  .. loglikelihood=-951.2722802086295	0.9988950059178572
Writing name finder model ... done (1.638s)

Wrote name finder model to
path: .\en_ner_person.bin
		</pre><p>
		</p>
		</div>
		<div class="section" title="Evaluating with English data"><div class="titlepage"><div><div><h4 class="title"><a name="tools.corpora.conll.2003.evaluation.english"></a>Evaluating with English data</h4></div></div></div>
		
		<p>
		Since we created the test A and B files above, we can use them to evaluate the model.
		</p><pre class="programlisting">
			
$ bin/opennlp TokenNameFinderEvaluator -lang en -encoding utf8 -model en_ner_person.bin \
    -data corpus_testa.txt
		</pre><p>
		</p><pre class="programlisting">
			
Loading Token Name Finder model ... done (0.359s)
current: 190.2 sent/s avg: 190.2 sent/s total: 199 sent
current: 648.3 sent/s avg: 415.9 sent/s total: 850 sent
current: 530.1 sent/s avg: 453.6 sent/s total: 1380 sent
current: 793.8 sent/s avg: 539.0 sent/s total: 2178 sent
current: 705.4 sent/s avg: 571.9 sent/s total: 2882 sent


Average: 569.4 sent/s
Total: 3251 sent
Runtime: 5.71s

Precision: 0.9366247297154147
Recall: 0.739956568946797
F-Measure: 0.8267557582133971
		</pre><p>
		</p>
		</div>
	</div>
	</div>
	<div class="section" title="Arvores Deitadas"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.corpora.arvores-deitadas"></a>Arvores Deitadas</h2></div></div></div>
		
		<p>
		The Portuguese corpora available at <a class="ulink" href="http://www.linguateca.pt" target="_top">Floresta Sint&aacute;(c)tica</a> project follow the Arvores Deitadas (AD) format. Apache OpenNLP includes tools to convert from AD format to native format.  
		</p>		
		<div class="section" title="Getting the data"><div class="titlepage"><div><div><h3 class="title"><a name="tools.corpora.arvores-deitadas.getting"></a>Getting the data</h3></div></div></div>
			
			<p>
			The Corpus can be downloaded from here: <a class="ulink" href="http://www.linguateca.pt/floresta/corpus.html" target="_top">http://www.linguateca.pt/floresta/corpus.html</a>
			</p>
			<p>
			The Name Finder models were trained using the Amazonia corpus: <a class="ulink" href="http://www.linguateca.pt/floresta/ficheiros/gz/amazonia.ad.gz" target="_top">amazonia.ad</a>.
			The Chunker models were trained using the <a class="ulink" href="http://www.linguateca.pt/floresta/ficheiros/gz/Bosque_CF_8.0.ad.txt.gz" target="_top">Bosque_CF_8.0.ad</a>.
			</p>
		</div>
		
		<div class="section" title="Converting the data"><div class="titlepage"><div><div><h3 class="title"><a name="tools.corpora.arvores-deitadas.converting"></a>Converting the data</h3></div></div></div>
			
			<p>
				To extract NameFinder training data from Amazonia corpus:
			</p><pre class="programlisting">
			
$ bin/opennlp TokenNameFinderConverter ad -encoding ISO-8859-1 -data amazonia.ad \
    -lang pt -types per &gt; corpus.txt
			</pre><p>
			</p>
			<p>
				To extract Chunker training data from Bosque_CF_8.0.ad corpus:
			</p><pre class="programlisting">
			
$ bin/opennlp ChunkerConverter ad -encoding ISO-8859-1 -data Bosque_CF_8.0.ad.txt &gt; bosque-chunk
			</pre><p>
			</p>
		</div>
		<div class="section" title="Evaluation"><div class="titlepage"><div><div><h3 class="title"><a name="tools.corpora.arvores-deitadas.evaluation"></a>Evaluation</h3></div></div></div>
			
			<p>
			To perform the evaluation the corpus was split into a training and a test part.
			</p><pre class="programlisting">
			
$ sed '1,55172d' corpus.txt &gt; corpus_train.txt
$ sed '55172,100000000d' corpus.txt &gt; corpus_test.txt
			</pre><p>
			</p><pre class="programlisting">
			
$ bin/opennlp TokenNameFinderTrainer -lang PT -encoding UTF-8 -data corpus_train.txt \
    -model pt-ner.bin -cutoff 20
..
$ bin/opennlp TokenNameFinderEvaluator -encoding UTF-8 -model ../model/pt-ner.bin \
    -data corpus_test.txt

Precision: 0.8005071889818507
Recall: 0.7450581122145297
F-Measure: 0.7717879983140168
			</pre><p>
			</p>
		</div>
	</div>
	<div class="section" title="Leipzig Corpora"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="tools.corpora.leipzig"></a>Leipzig Corpora</h2></div></div></div>
	
	<p>
	The Leiopzig Corpora collection presents corpora in different languages. The corpora is a collection of individual sentences collected
	from the web and newspapers. The Corpora is available as plain text and as MySQL database tables. The OpenNLP integration can only
	use the plain text version.
	</p>
	<p>
	The corpora in the different languages can be used to train a document categorizer model which can detect the document language. 
	The	individual plain text packages can be downlaoded here:
	<a class="ulink" href="http://corpora.uni-leipzig.de/download.html" target="_top">http://corpora.uni-leipzig.de/download.html</a>
	</p>
	
	<p>
	Afer all packages have been downloaded, unzip them and use the following commands to
	produce a training file which can be processed by the Document Categorizer:
	</p><pre class="programlisting">
			
bin/opennlp DoccatConverter leipzig -lang cat -data Leipzig/cat100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang de -data Leipzig/de100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang dk -data Leipzig/dk100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang ee -data Leipzig/ee100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang en -data Leipzig/en100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang fi -data Leipzig/fi100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang fr -data Leipzig/fr100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang it -data Leipzig/it100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang jp -data Leipzig/jp100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang kr -data Leipzig/kr100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang nl -data Leipzig/nl100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang no -data Leipzig/no100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang se -data Leipzig/se100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang sorb -data Leipzig/sorb100k/sentences.txt &gt;&gt; lang.train
bin/opennlp DoccatConverter leipzig -lang tr -data Leipzig/tr100k/sentences.txt &gt;&gt; lang.train
	</pre><p>
	</p>
	<p>
	Depending on your platform local it might be problemmatic to output characters which are not supported by that encoding,
	we suggest to run these command on a platform which has a unicode default encoding, e.g. Linux with UTF-8.
	</p>
	<p>
	Afer the lang.train file is created the actual language detection document categorizer model
	can be created with the following command.
	</p><pre class="programlisting">
			
bin/opennlp DoccatTrainer -lang x-unspecified -encoding MacRoman -data ../lang.train -model lang.model
Indexing events using cutoff of 5

	Computing event counts...  done. 10000 events
	Indexing...  done.
Sorting and merging events... done. Reduced 10000 events to 10000.
Done indexing.
Incorporating indexed data for training...  
done.
	Number of Event Tokens: 10000
	    Number of Outcomes: 2
	  Number of Predicates: 42730
...done.
Computing model parameters...
Performing 100 iterations.
  1:  .. loglikelihood=-6931.471805600547	0.5
  2:  .. loglikelihood=-2110.9654348555955	1.0
... cut lots of iterations ...

 99:  .. loglikelihood=-0.449640418555347	1.0
100:  .. loglikelihood=-0.443746359746235	1.0
Writing document categorizer model ... done (1.210s)

Wrote document categorizer model to
path: /Users/joern/dev/opennlp-apache/opennlp/opennlp-tools/lang.model

	</pre><p>
	In the sample above the language detection model was trained to distinguish two languages, danish and english.
	</p>
	
	<p>
	After the model is created it can be used to detect the two languages:
	
	</p><pre class="programlisting">
			
$ bin/opennlp Doccat ../lang.
lang.model  lang.train  
karkand:opennlp-tools joern$ bin/opennlp Doccat ../lang.model
Loading Document Categorizer model ... done (0.289s)
The American Finance Association is pleased to announce the award of ...
en	The American Finance Association is pleased to announce the award of ..
.
Danskerne skal betale for den &oslash;konomiske krise ved at blive l&aelig;ngere p&aring; arbejdsmarkedet .
dk	Danskerne skal betale for den &oslash;konomiske krise ved at blive l&aelig;ngere p&aring; arbejdsmarkedet .	
	</pre><p>
	</p>
	</div>
</div>
	<div class="chapter" title="Chapter&nbsp;11.&nbsp;Machine Learning"><div class="titlepage"><div><div><h2 class="title"><a name="opennlp.ml"></a>Chapter&nbsp;11.&nbsp;Machine Learning</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#opennlp.ml.maxent">Maximum Entropy</a></span></dt><dd><dl><dt><span class="section"><a href="#opennlp.ml.maxent.impl">Implementation</a></span></dt></dl></dd></dl></div>

	<div class="section" title="Maximum Entropy"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="opennlp.ml.maxent"></a>Maximum Entropy</h2></div></div></div>
		
		<p>
		To explain what maximum entropy is, it will be simplest to quote from Manning and Schutze* (p. 589):
		<span class="quote">&#8220;<span class="quote">
		Maximum entropy modeling is a framework for integrating information from many heterogeneous
		information sources for classification.  The data for a  classification problem is described
		as a (potentially large) number of features.  These features can be quite complex and allow
		the experimenter to make use of prior knowledge about what types of informations are expected
		to be important for classification. Each feature corresponds to a constraint on the model.
		We then compute the maximum entropy model, the model with the maximum entropy of all the models
		that satisfy the constraints.  This term may seem perverse, since we have spent most of the book
		trying to minimize the (cross) entropy of models, but the idea is that we do not want to go beyond
		the data.  If we chose a model with less entropy, we would add `information' constraints to the
		model that are not justified by the empirical evidence available to us. Choosing the maximum
		entropy model is motivated by the desire to preserve as much uncertainty as possible.
		</span>&#8221;</span>
		</p>
		<p>
		So that gives a rough idea of what the maximum entropy framework is.
		Don't assume anything about your probability distribution other than what you have observed. 
		</p>
		<p>
		On the engineering level, using maxent is an excellent way of creating programs which perform
		very difficult classification tasks very well.  For example,  precision and recall figures for
		programs using maxent models have reached (or are) the state of the art on tasks like part of
		speech tagging, sentence detection, prepositional phrase attachment, and named entity recognition.
		On the engineering level, an added benefit is that the person creating a maxent model only needs
		to inform the training procedure of the event space, and need not worry about independence between
		features.
		</p>
		<p>
		While the authors of this implementation of maximum entropy are generally interested using
		maxent models in natural language processing, the framework is certainly quite general and
		useful for a much wider variety of fields.  In fact, maximum entropy modeling was originally
		developed for statistical physics.
		</p>
		<p>
		For a very in-depth discussion of how maxent can be used in natural language processing,
		try reading Adwait Ratnaparkhi's dissertation.   Also,  check out Berger, Della Pietra,
		and Della Pietra's paper A Maximum Entropy Approach to Natural Language Processing, which
		provides an excellent introduction and discussion of the framework.
		</p>
		<p>
		*Foundations of statistical natural language processing . Christopher D. Manning, Hinrich Schutze. 
		Cambridge, Mass. : MIT Press, c1999.
		</p>
			<div class="section" title="Implementation"><div class="titlepage"><div><div><h3 class="title"><a name="opennlp.ml.maxent.impl"></a>Implementation</h3></div></div></div>
		
		<p>
		We have tried to make the opennlp.maxent implementation easy to use.  To create a model, one
		needs (of course) the training data, and then implementations of two interfaces in the
		opennlp.maxent package, EventStream and ContextGenerator.  These have fairly simple specifications,
		and example implementations can be found in the OpenNLP Tools preprocessing components.
		</p>
		<p>
		We have also set in place some interfaces and code to make it easier to automate the training
		and evaluation process (the Evalable interface and the TrainEval class).  It is not necessary
		to use this functionality, but if you do you'll find it much easier to see how well your models
		are doing.  The opennlp.grok.preprocess.namefind package is an example of a maximum entropy
		component which uses this functionality.
		</p>
		<p>
		We have managed to use several techniques to reduce the size of the models when writing them to
		disk, which also means that reading in a model for use is much quicker than with less compact
		encodings of the model.  This was especially important to us since we use many maxent models in
		the Grok library, and we wanted the start up time and the physical size of the library to be as
		minimal as possible. As of version 1.2.0, maxent has an io package which greatly simplifies the
		process of loading and saving models in different formats.
		</p>
		</div>
	</div>
</div>
	<div class="chapter" title="Chapter&nbsp;12.&nbsp;UIMA Integration"><div class="titlepage"><div><div><h2 class="title"><a name="org.apche.opennlp.uima"></a>Chapter&nbsp;12.&nbsp;UIMA Integration</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="section"><a href="#org.apche.opennlp.running-pear-sample">Running the pear sample in CVD</a></span></dt><dt><span class="section"><a href="#org.apche.opennlp.further-help">Further Help</a></span></dt></dl></div>

<p>
	The UIMA Integration wraps the OpenNLP components in UIMA Analysis Engines which can 
	be used to automatically annotate text and train new OpenNLP models from annotated text.
</p>
	<div class="section" title="Running the pear sample in CVD"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="org.apche.opennlp.running-pear-sample"></a>Running the pear sample in CVD</h2></div></div></div>
		
		<p>
			The Cas Visual Debugger is shipped as part of the UIMA distribution and is a tool which can run
			the OpenNLP UIMA Annotators and display their analysis results. The source distribution comes with a script
			which can create a sample UIMA application. Which includes the sentence detector, tokenizer,
			pos tagger, chunker and name finders for English. This sample application is packaged in the
			pear format and must be installed with the pear installer before it can be run by CVD.
			Please consult the UIMA documentation for further information about the pear installer.
		</p>
		<p>
			The OpenNLP UIMA pear file must be build manually.
			First download the source distribution, unzip it and go to the apache-opennlp/opennlp folder.
			Type "mvn install" to build everything. Now build the pear file, go to apache-opennlp/opennlp-uima
			and build it as shown below. Note the models will be downloaded
			from the old SourceForge repository and are not licensed under the AL 2.0.
			</p><pre class="screen">
			
$ ant -f createPear.xml 
Buildfile: createPear.xml

createPear:
     [echo] ##### Creating OpenNlpTextAnalyzer pear #####
     [copy] Copying 13 files to OpenNlpTextAnalyzer/desc
     [copy] Copying 1 file to OpenNlpTextAnalyzer/metadata
     [copy] Copying 1 file to OpenNlpTextAnalyzer/lib
     [copy] Copying 3 files to OpenNlpTextAnalyzer/lib
    [mkdir] Created dir: OpenNlpTextAnalyzer/models
      [get] Getting: http://opennlp.sourceforge.net/models-1.5/en-token.bin
      [get] To: OpenNlpTextAnalyzer/models/en-token.bin
      [get] Getting: http://opennlp.sourceforge.net/models-1.5/en-sent.bin
      [get] To: OpenNlpTextAnalyzer/models/en-sent.bin
      [get] Getting: http://opennlp.sourceforge.net/models-1.5/en-ner-date.bin
      [get] To: OpenNlpTextAnalyzer/models/en-ner-date.bin
      [get] Getting: http://opennlp.sourceforge.net/models-1.5/en-ner-location.bin
      [get] To: OpenNlpTextAnalyzer/models/en-ner-location.bin
      [get] Getting: http://opennlp.sourceforge.net/models-1.5/en-ner-money.bin
      [get] To: OpenNlpTextAnalyzer/models/en-ner-money.bin
      [get] Getting: http://opennlp.sourceforge.net/models-1.5/en-ner-organization.bin
      [get] To: OpenNlpTextAnalyzer/models/en-ner-organization.bin
      [get] Getting: http://opennlp.sourceforge.net/models-1.5/en-ner-percentage.bin
      [get] To: OpenNlpTextAnalyzer/models/en-ner-percentage.bin
      [get] Getting: http://opennlp.sourceforge.net/models-1.5/en-ner-person.bin
      [get] To: OpenNlpTextAnalyzer/models/en-ner-person.bin
      [get] Getting: http://opennlp.sourceforge.net/models-1.5/en-ner-time.bin
      [get] To: OpenNlpTextAnalyzer/models/en-ner-time.bin
      [get] Getting: http://opennlp.sourceforge.net/models-1.5/en-pos-maxent.bin
      [get] To: OpenNlpTextAnalyzer/models/en-pos-maxent.bin
      [get] Getting: http://opennlp.sourceforge.net/models-1.5/en-chunker.bin
      [get] To: OpenNlpTextAnalyzer/models/en-chunker.bin
      [zip] Building zip: OpenNlpTextAnalyzer.pear

BUILD SUCCESSFUL
Total time: 3 minutes 20 seconds
		 </pre><p>
		</p>
		<p>
			After the pear is installed start the Cas Visual Debugger shipped with the UIMA framework.
			And click on Tools -&gt; Load AE. Then select the opennlp.uima.OpenNlpTextAnalyzer_pear.xml
			file in the file dialog. Now enter some text and start the analysis engine with
			"Run -&gt; Run OpenNLPTextAnalyzer". Afterwards the results will be displayed.
			You should see sentences, tokens, chunks, pos tags and maybe some names. Remember the input text
			must be written in English.
		</p>
	</div>
	<div class="section" title="Further Help"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="org.apche.opennlp.further-help"></a>Further Help</h2></div></div></div>
		
		<p>
			For more information about how to use the integration please consult the javadoc of the individual
			Analysis Engines and checkout the included xml descriptors.
		</p>
		<p>
			TODO: Extend this documentation with information about the individual components.
			If you want to contribute please contact us on the mailing list
			or comment on the jira issue <a class="ulink" href="https://issues.apache.org/jira/browse/OPENNLP-49" target="_top">OPENNLP-49</a>.
		</p>
	</div>
</div>
</div></body></html>